| Age | Commit message (Collapse) | Author |
|---|
|
Rename the newly added arm barriers and READ/WRITE_ONCE
to avoid using popular names.
Signed-off-by: Julia Suvorova <jusual@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The names of the barriers conflict with the namespaces of other projects
when trying to directly include liburing.h. Avoid using popular global
names.
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Julia Suvorova <jusual@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
making liburing on arm64 platform failed. let's support it.
root@Kylin:/# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux buster/sid"
root@Kylin:/# gcc -v
gcc version 8.2.0 (Debian 8.2.0-20)
root@Kylin:~/liburing# make
make[1]: Entering directory '/root/liburing/src'
cc -g -fomit-frame-pointer -O2 -Wall -Iinclude/ -c -o setup.ol setup.c
In file included from include/liburing.h:14,
from setup.c:10:
include/liburing.h: In function 'io_uring_cq_advance':
include/liburing/barrier.h:73:2: warning: implicit declaration of function 'smp_mb'; did you mean 'smp_wmb'? [-Wimplicit-function-declaration]
smp_mb(); \
^~~~~~
include/liburing.h:111:3: note: in expansion of macro 'smp_store_release'
smp_store_release(cq->khead, *cq->khead + nr);
^~~~~~~~~~~~~~~~~
Signed-off-by: Jackie Liu <liuyun01@kylinos.cn>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
It is not possible to install barrier.h and compat.h into the top-level
/usr/include directly since they are likely to conflict with other
software. io_uring.h could be confused with the system's kernel header
file.
Put liburing headers into <liburing/*.h> so there is no chance of
conflicts or confusion.
Existing applications continue to build successfully since the location
of <liburing.h> is unchanged. In-tree examples and tests require
modification because src/liburing.h is moved to src/include/liburing.h.
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The top-level makefile passes in includedir and libdir so we should
not override them.
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
typeof() is not available on all compilers. __typeof() is
a safer bet.
Signed-off-by: Hrvoje Zeba <zeba.hrvoje@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Use identical memory barrier implementations on 32 and 64 bit Intel CPUs.
In the past the Linux kernel supported 32 bit CPUs that violate the x86
ordering standard. Since io_uring is not supported by these older kernels,
do not support these older CPUs in liburing. See also Linux kernel commit
5927145efd5d ("x86/cpu: Remove the CONFIG_X86_PPRO_FENCE=y quirk") # v4.16.
Cc: Roman Penyaev <rpenyaev@suse.de>
Suggested-by: Roman Penyaev <rpenyaev@suse.de>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
This patch improves consistency with the Linux kernel source code.
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
This patch makes the liburing build work as expected for e.g. the following
command:
make CFLAGS=-m32
and avoids that the build fails as follows for the above command:
make[1]: Entering directory 'liburing/test'
cc -m32 -o poll poll.c -luring
/usr/bin/ld: cannot find -luring
collect2: error: ld returned 1 exit status
Makefile:18: recipe for target 'poll' failed
make[1]: *** [poll] Error 1
make[1]: Leaving directory 'software/liburing/test'
Makefile:12: recipe for target 'all' failed
make: *** [all] Error 2
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Introduce the smp_load_acquire() and smp_store_release() macros. Fix
synchronization in io_uring_cq_advance() and __io_uring_get_cqe().
Remove a superfluous local variable, if-test and write barrier from
__io_uring_submit(). Remove a superfluous barrier from
test/io_uring_enter.c.
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Roman Penyaev <rpenyaev@suse.de>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Use io_uring_for_each_cqe() inside __io_uring_get_cqe() such that it
becomes possible to test the io_uring_for_each_cqe() implementation
from inside the liburing project.
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Roman Penyaev <rpenyaev@suse.de>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Don't check for IORING_SQ_NEED_WAKEUP twice, put it inside
sq_ring_needs_enter().
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
We don't modify them, make that clear.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Works just like io_uring_submit(), but also allows retrieving events
(or waiting/polling for them) in the same call.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Ring memory is released by umapping {cq,sq}->khead, which assumes that
p->{cq,sq}_off.head will always be 0.
Add another field to for the mmaped region and use that instead of
->khead when umapping.
Signed-off-by: Kornilios Kourtis <kkourt@kkourt.io>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_uring_queue_mmap wasn't exported from the ld map file, meaning the
linker gave it private, static linkage in the resulting shared object,
as opposed to global linkage. This results in the symbol seemingly
missing at link-time, despite the existence of the symbol in the library
itself.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_uring_cqe_seen() contains write memory barrier (at least for !x86),
so calling it can be very expensive for a big number of cqes.
Would be better to have a loop over all available cqes and only then
advance the cq ring once, e.g.:
struct io_uring_cqe *cqe;
unsigned nr, head;
nr = 0;
io_uring_for_each_cqe(&ring, head, cqe) {
/* handle cqe */
if (++nr == max)
break;
}
io_uring_cq_advance(&ring, nr);
Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_uring_submit() is a last commit point when a caller has to be sure
that all prepared sqes eventually lands to the kernel.
If SQ thread is running it is quite possible that khead != ktail, thus
charging of sq->array is skipped, which leads to IO hang.
Nasty hang is well reproduced when submitter and completion harverster
are different threads.
Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Enter kernel only if SQ thread is off or wakeup is needed.
Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
This syncs with io_uring-next, which has support for doing recvmsg
and sendmsg through io_uring.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Use proper unsigned math to figure out how many entries we have. If
we have head and tail on either side of UINT_MAX, then we currently
don't submit anything as:
while (head < tail) {
is never true.
Reported-by: Weiping Zhang <zhangweiping@didiglobal.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
If the ring setup fails, close the fd before returning failure.
Reported-by: Kornilios Kourtis <kkourt@kkourt.io>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
We really should use a ';' to terminate the line. This is just a
cosmetic issue, no functional changes.
Signed-off-by: Weiping Zhang <zhangweiping@didiglobal.com>
Modified wording
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Some of the symbols were missing from linker version
script so they weren't exported in the dynamic library.
Signed-off-by: Hrvoje Zeba <zeba.hrvoje@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Fortunately, all architectures except alpha have common numbers
for new system calls nowadays, so support of non-alpha architectures
is as simple as support of x86.
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Rename __NR_sys_io_uring* to __NR_io_uring* to match kernel headers.
The prefix traditionally used for macros describing syscall numbers
is "__NR_", not "__NR_sys_".
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_uring_prep_{readv,writev,read_fixed,write_fixed}() take
non-const arguments that are const in the native functions
that they mimic. Make those arguments const.
Signed-off-by: Ian Gulliver <git@flamingcow.io>
|
|
liburing.h uses memset() but doesn't include string.h.
If nothing else includes it, this causes compile errors.
Add the include.
Signed-off-by: Ian Gulliver <git@flamingcow.io>
|
|
Add sync_range_file opcode and related fields, and the
io_uring_register(2) commands for eventfd registration.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Just a basic test case that does various forms of linked nops, and
a sample bare bones copy program using linked reads and writes.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Make header includable from C++ projects.
Signed-off-by: Kevin Vigor <kvigor@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Static libraries are generally built without -fPIC for slight
performance gain, make that the case for liburing.
Signed-off-by: Kevin Vigor <kvigor@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
'make install' did not install all the headers necessary to use the
library; fix.
Additionally allow specifying the install prefix, which was previously
hardcoded to /usr.
Signed-off-by: Kevin Vigor <kvigor@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
We have io_uring_get_sqe() on the submission side, yet the completion
side is named _completion. Rename as follows:
io_uring_get_completion() io_uring_peek_cqe()
iO_uring_wait_completion() io_uring_wait_cqe()
This better tells the user what the _get variant does by calling it
_peek instead, and we move to using _cqe() as the postfix instead
of _completion.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
There's a failure case where an application gets a cqe entry, but
the kernel can then overwrite it before the application is done
reading it. This can happen since the io_uring_{get,wait}_completion()
interface both returns a CQE pointer AND increments the ring index.
If the kernel reuses this entry before the applications is done reading
it, the contents may be corrupted.
Remove the CQ head increment from the CQE retrieval, and put it into
a separate helper, io_uring_cqe_seen(). The application must call this
helper when it got a new CQE entry through one of the above calls, and
it's now done reading it.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
I generalized the barrier flag to be applicable to all commands, so
let's drop the fsync special flag.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Signed-off-by: Zach Bjornson <zbbjornson@gmail.com>
Add uintptr_t cast to avoid 32-bit warnings.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Also changes the fsync prep helper to require passing in the actual
flag, not just a boolean for fsync vs fdatasync.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
This cache hint has been removed from the io_uring series, as there's
some hesitation to leak this information to userspace. Remove it from
the header file and the man page. We can always re-introduce it later,
if we get some variant of this included.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
A previous fix that ensured we pass back the right error messed
up the normal return, which is number of entries submitted.
This makes the poll test cases fail.
Fixes: 8260029608b9 ("queue: ensure io_uring_submit() returns the right error")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
We weren't passing back -errno for the system call failure.
This meant any error got turned into EPERM as far as the
caller was concerned.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Also unify the setup, all the read/write variants can use the same
helper.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
This really wants to be a "will we over-fill the ring?" kind of
check, but the sqe_head/sqe_tail should not be that far apart. If they
are, that's a bug elsewhere. So just kill the check.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
