summaryrefslogtreecommitdiff
path: root/src/queue.c
AgeCommit message (Collapse)Author
2019-07-02Fix the use of memory barriersBart Van Assche
Introduce the smp_load_acquire() and smp_store_release() macros. Fix synchronization in io_uring_cq_advance() and __io_uring_get_cqe(). Remove a superfluous local variable, if-test and write barrier from __io_uring_submit(). Remove a superfluous barrier from test/io_uring_enter.c. Cc: Stefan Hajnoczi <stefanha@redhat.com> Cc: Roman Penyaev <rpenyaev@suse.de> Signed-off-by: Bart Van Assche <bvanassche@acm.org> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-07-02__io_uring_get_cqe(): Use io_uring_for_each_cqe()Bart Van Assche
Use io_uring_for_each_cqe() inside __io_uring_get_cqe() such that it becomes possible to test the io_uring_for_each_cqe() implementation from inside the liburing project. Cc: Stefan Hajnoczi <stefanha@redhat.com> Cc: Roman Penyaev <rpenyaev@suse.de> Signed-off-by: Bart Van Assche <bvanassche@acm.org> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-17src/queue: cleanup sq_ring_needs_enter()Jens Axboe
Don't check for IORING_SQ_NEED_WAKEUP twice, put it inside sq_ring_needs_enter(). Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-06Add io_uring_submit_and_wait()Jens Axboe
Works just like io_uring_submit(), but also allows retrieving events (or waiting/polling for them) in the same call. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-27queue: always fill in sq->array on io_uring_submit()Roman Penyaev
io_uring_submit() is a last commit point when a caller has to be sure that all prepared sqes eventually lands to the kernel. If SQ thread is running it is quite possible that khead != ktail, thus charging of sq->array is skipped, which leads to IO hang. Nasty hang is well reproduced when submitter and completion harverster are different threads. Signed-off-by: Roman Penyaev <rpenyaev@suse.de> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-27liburing,queue,setup: handle IORING_SQ_NEED_WAKEUP for io_uring_submit()Roman Penyaev
Enter kernel only if SQ thread is off or wakeup is needed. Signed-off-by: Roman Penyaev <rpenyaev@suse.de> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-05-21io_uring_submit: fix head/tail wrap issueJens Axboe
Use proper unsigned math to figure out how many entries we have. If we have head and tail on either side of UINT_MAX, then we currently don't submit anything as: while (head < tail) { is never true. Reported-by: Weiping Zhang <zhangweiping@didiglobal.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-18Rename completion helpersJens Axboe
We have io_uring_get_sqe() on the submission side, yet the completion side is named _completion. Rename as follows: io_uring_get_completion() io_uring_peek_cqe() iO_uring_wait_completion() io_uring_wait_cqe() This better tells the user what the _get variant does by calling it _peek instead, and we move to using _cqe() as the postfix instead of _completion. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-17Add io_uring_cqe_seen()Jens Axboe
There's a failure case where an application gets a cqe entry, but the kernel can then overwrite it before the application is done reading it. This can happen since the io_uring_{get,wait}_completion() interface both returns a CQE pointer AND increments the ring index. If the kernel reuses this entry before the applications is done reading it, the contents may be corrupted. Remove the CQ head increment from the CQE retrieval, and put it into a separate helper, io_uring_cqe_seen(). The application must call this helper when it got a new CQE entry through one of the above calls, and it's now done reading it. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-03-13queue: ensure io_uring_submit() returns the system call valueJens Axboe
A previous fix that ensured we pass back the right error messed up the normal return, which is number of entries submitted. This makes the poll test cases fail. Fixes: 8260029608b9 ("queue: ensure io_uring_submit() returns the right error") Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-03-05queue: ensure io_uring_submit() returns the right errorJens Axboe
We weren't passing back -errno for the system call failure. This meant any error got turned into EPERM as far as the caller was concerned. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-02-15src/queue: kill bogus ktail_next == head checkJens Axboe
This really wants to be a "will we over-fill the ring?" kind of check, but the sqe_head/sqe_tail should not be that far apart. If they are, that's a bug elsewhere. So just kill the check. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-02-10src/queue: add comments on the read and write barriersJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-02-08io_uring_enter: don't expose sigset_size argumentJens Axboe
Applications should not need to care about this, we can pass it in ourselves. Once the libc support is there, we won't expose this parameter either. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-02-08Add sigmask parameter to io_uring_enterJeff Moyer
Update liburing and io_uring_enter.2 to match the kernel. Signed-off-by: Jeff Moyer <jmoyer@redhat.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-17Split src/io_uring.c upJens Axboe
Let's have the various helpers be in usefully named functions, no need to bundle them all into the same one. Signed-off-by: Jens Axboe <axboe@kernel.dk>