summaryrefslogtreecommitdiff
path: root/src
AgeCommit message (Collapse)Author
2019-07-02Fix the use of memory barriersBart Van Assche
Introduce the smp_load_acquire() and smp_store_release() macros. Fix synchronization in io_uring_cq_advance() and __io_uring_get_cqe(). Remove a superfluous local variable, if-test and write barrier from __io_uring_submit(). Remove a superfluous barrier from test/io_uring_enter.c. Cc: Stefan Hajnoczi <stefanha@redhat.com> Cc: Roman Penyaev <rpenyaev@suse.de> Signed-off-by: Bart Van Assche <bvanassche@acm.org> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-07-02__io_uring_get_cqe(): Use io_uring_for_each_cqe()Bart Van Assche
Use io_uring_for_each_cqe() inside __io_uring_get_cqe() such that it becomes possible to test the io_uring_for_each_cqe() implementation from inside the liburing project. Cc: Stefan Hajnoczi <stefanha@redhat.com> Cc: Roman Penyaev <rpenyaev@suse.de> Signed-off-by: Bart Van Assche <bvanassche@acm.org> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-17src/queue: cleanup sq_ring_needs_enter()Jens Axboe
Don't check for IORING_SQ_NEED_WAKEUP twice, put it inside sq_ring_needs_enter(). Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-17Add support for eventfd registrationJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-17Make io_uring registration arguments constJens Axboe
We don't modify them, make that clear. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-06Add io_uring_submit_and_wait()Jens Axboe
Works just like io_uring_submit(), but also allows retrieving events (or waiting/polling for them) in the same call. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-04Add basic helpers for file/buffer registrationJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-03Do not assume p->{cq,sq}_off.head is 0Kornilios Kourtis
Ring memory is released by umapping {cq,sq}->khead, which assumes that p->{cq,sq}_off.head will always be 0. Add another field to for the mmaped region and use that instead of ->khead when umapping. Signed-off-by: Kornilios Kourtis <kkourt@kkourt.io> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-29liburing: export io_uring_queue_mmap symbolAustin Seipp
io_uring_queue_mmap wasn't exported from the ld map file, meaning the linker gave it private, static linkage in the resulting shared object, as opposed to global linkage. This results in the symbol seemingly missing at link-time, despite the existence of the symbol in the library itself. Signed-off-by: Austin Seipp <aseipp@pobox.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-27liburing: introduce io_uring_for_each_cqe() and io_uring_cq_advance()Roman Penyaev
io_uring_cqe_seen() contains write memory barrier (at least for !x86), so calling it can be very expensive for a big number of cqes. Would be better to have a loop over all available cqes and only then advance the cq ring once, e.g.: struct io_uring_cqe *cqe; unsigned nr, head; nr = 0; io_uring_for_each_cqe(&ring, head, cqe) { /* handle cqe */ if (++nr == max) break; } io_uring_cq_advance(&ring, nr); Signed-off-by: Roman Penyaev <rpenyaev@suse.de> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-27liburing: introduce io_uring_cqe_set_flags() helperRoman Penyaev
Signed-off-by: Roman Penyaev <rpenyaev@suse.de> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-27queue: always fill in sq->array on io_uring_submit()Roman Penyaev
io_uring_submit() is a last commit point when a caller has to be sure that all prepared sqes eventually lands to the kernel. If SQ thread is running it is quite possible that khead != ktail, thus charging of sq->array is skipped, which leads to IO hang. Nasty hang is well reproduced when submitter and completion harverster are different threads. Signed-off-by: Roman Penyaev <rpenyaev@suse.de> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-27liburing,queue,setup: handle IORING_SQ_NEED_WAKEUP for io_uring_submit()Roman Penyaev
Enter kernel only if SQ thread is off or wakeup is needed. Signed-off-by: Roman Penyaev <rpenyaev@suse.de> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-27setup: return -errno explicitly from io_uring_queue_init()Roman Penyaev
Signed-off-by: Roman Penyaev <rpenyaev@suse.de> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-23io_uring.h: add send/recvmsg partsJens Axboe
This syncs with io_uring-next, which has support for doing recvmsg and sendmsg through io_uring. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-21io_uring_submit: fix head/tail wrap issueJens Axboe
Use proper unsigned math to figure out how many entries we have. If we have head and tail on either side of UINT_MAX, then we currently don't submit anything as: while (head < tail) { is never true. Reported-by: Weiping Zhang <zhangweiping@didiglobal.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-20Fix leak of fd in io_uring_queue_init() on failureJens Axboe
If the ring setup fails, close the fd before returning failure. Reported-by: Kornilios Kourtis <kkourt@kkourt.io> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-20fix harmless typo in sq_entries in io_uring_mmapWeiping Zhang
We really should use a ';' to terminate the line. This is just a cosmetic issue, no functional changes. Signed-off-by: Weiping Zhang <zhangweiping@didiglobal.com> Modified wording Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-20Add missing symbols to linker version mapHrvoje Zeba
Some of the symbols were missing from linker version script so they weren't exported in the dynamic library. Signed-off-by: Hrvoje Zeba <zeba.hrvoje@gmail.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-20syscall: add support of non-x86 architecturesDmitry V. Levin
Fortunately, all architectures except alpha have common numbers for new system calls nowadays, so support of non-alpha architectures is as simple as support of x86. Signed-off-by: Dmitry V. Levin <ldv@altlinux.org> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-20syscall: fix names of __NR_* macrosDmitry V. Levin
Rename __NR_sys_io_uring* to __NR_io_uring* to match kernel headers. The prefix traditionally used for macros describing syscall numbers is "__NR_", not "__NR_sys_". Signed-off-by: Dmitry V. Levin <ldv@altlinux.org> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-19Match const signatures in io_uring_prep_* to native functionsIan Gulliver
io_uring_prep_{readv,writev,read_fixed,write_fixed}() take non-const arguments that are const in the native functions that they mimic. Make those arguments const. Signed-off-by: Ian Gulliver <git@flamingcow.io>
2019-05-18Fix liburing.h memset() missing declarationIan Gulliver
liburing.h uses memset() but doesn't include string.h. If nothing else includes it, this causes compile errors. Add the include. Signed-off-by: Ian Gulliver <git@flamingcow.io>
2019-05-18io_uring.h: sync with kernelJens Axboe
Add sync_range_file opcode and related fields, and the io_uring_register(2) commands for eventfd registration. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-13Add link SQE supportJens Axboe
Just a basic test case that does various forms of linked nops, and a sample bare bones copy program using linked reads and writes. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-06liburing: add extern "C" to liburing.hKevin Vigor
Make header includable from C++ projects. Signed-off-by: Kevin Vigor <kvigor@gmail.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-06liburing: do not build static library with -fPICKevin Vigor
Static libraries are generally built without -fPIC for slight performance gain, make that the case for liburing. Signed-off-by: Kevin Vigor <kvigor@gmail.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-06liburing: improve 'make install'Kevin Vigor
'make install' did not install all the headers necessary to use the library; fix. Additionally allow specifying the install prefix, which was previously hardcoded to /usr. Signed-off-by: Kevin Vigor <kvigor@gmail.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-18Rename completion helpersJens Axboe
We have io_uring_get_sqe() on the submission side, yet the completion side is named _completion. Rename as follows: io_uring_get_completion() io_uring_peek_cqe() iO_uring_wait_completion() io_uring_wait_cqe() This better tells the user what the _get variant does by calling it _peek instead, and we move to using _cqe() as the postfix instead of _completion. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-17Add io_uring_cqe_seen()Jens Axboe
There's a failure case where an application gets a cqe entry, but the kernel can then overwrite it before the application is done reading it. This can happen since the io_uring_{get,wait}_completion() interface both returns a CQE pointer AND increments the ring index. If the kernel reuses this entry before the applications is done reading it, the contents may be corrupted. Remove the CQ head increment from the CQE retrieval, and put it into a separate helper, io_uring_cqe_seen(). The application must call this helper when it got a new CQE entry through one of the above calls, and it's now done reading it. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-11test/nop: add NOP test caseJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-10test/fsync: update for IOSQE_IO_DRAIN approachJens Axboe
I generalized the barrier flag to be applicable to all commands, so let's drop the fsync special flag. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-07Add io_uring_cqe_get_data()Zach Bjornson
Signed-off-by: Zach Bjornson <zbbjornson@gmail.com> Add uintptr_t cast to avoid 32-bit warnings. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-06Add barrier fsync test caseJens Axboe
Also changes the fsync prep helper to require passing in the actual flag, not just a boolean for fsync vs fdatasync. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-03-19Remove IOCQE_FLAG_CACHEHITJens Axboe
This cache hint has been removed from the io_uring series, as there's some hesitation to leak this information to userspace. Remove it from the header file and the man page. We can always re-introduce it later, if we get some variant of this included. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-03-13queue: ensure io_uring_submit() returns the system call valueJens Axboe
A previous fix that ensured we pass back the right error messed up the normal return, which is number of entries submitted. This makes the poll test cases fail. Fixes: 8260029608b9 ("queue: ensure io_uring_submit() returns the right error") Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-03-05queue: ensure io_uring_submit() returns the right errorJens Axboe
We weren't passing back -errno for the system call failure. This meant any error got turned into EPERM as far as the caller was concerned. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-02-28Add read/write fixed prep supportJens Axboe
Also unify the setup, all the read/write variants can use the same helper. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-02-15src/queue: kill bogus ktail_next == head checkJens Axboe
This really wants to be a "will we over-fill the ring?" kind of check, but the sqe_head/sqe_tail should not be that far apart. If they are, that's a bug elsewhere. So just kill the check. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-02-10Sync io_uring.h with the kernelJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-02-10src/queue: add comments on the read and write barriersJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-02-08test/fsync: add simple fsync testerJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-02-08add io_uring_queue_mmapJeff Moyer
io_uring_queue_init does not allow the caller to specify sq_thread_cpu or sq_thread_idle. Users that want to specify those parameters need to call io_uring_setup(2) themselves. Add a helper so that they don't also have to hand-craft the code to map the submission and completion queues, and setup the sqe ring. This allows those applications to still make use of io_uring_submit and get/wait_completion. Signed-off-by: Jeff Moyer <jmoyer@redhat.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-02-08io_uring_enter: don't expose sigset_size argumentJens Axboe
Applications should not need to care about this, we can pass it in ourselves. Once the libc support is there, we won't expose this parameter either. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-02-08Add sigmask parameter to io_uring_enterJeff Moyer
Update liburing and io_uring_enter.2 to match the kernel. Signed-off-by: Jeff Moyer <jmoyer@redhat.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-02-08syscall: update io_uring_enter() to match kernel typeJens Axboe
We would randomly fail depending on what garbage was passed in the two sigset related fields. Reported-by: Jeff Moyer <jmoyer@redhat.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-02-05io_uring_register takes 4 argsJeff Moyer
Signed-off-by: Jeff Moyer <jmoyer@redhat.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-31src/io_uring.h: sync with kernel headerJens Axboe
Just a removal of the IORING_MAX_ENTRIES user visible define. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-29Update kernel io_uring headerJens Axboe
- struct io_uring_params changed (u16 -> u32 for sq options) - Use unsigned for the flag types Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-23System calls have been renumberedJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
generated by cgit (git 2.34.1) at 2024-11-14 17:48:58 +0000