| Age | Commit message (Collapse) | Author |
|---|
|
Introduce the smp_load_acquire() and smp_store_release() macros. Fix
synchronization in io_uring_cq_advance() and __io_uring_get_cqe().
Remove a superfluous local variable, if-test and write barrier from
__io_uring_submit(). Remove a superfluous barrier from
test/io_uring_enter.c.
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Roman Penyaev <rpenyaev@suse.de>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Use io_uring_for_each_cqe() inside __io_uring_get_cqe() such that it
becomes possible to test the io_uring_for_each_cqe() implementation
from inside the liburing project.
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Roman Penyaev <rpenyaev@suse.de>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Don't check for IORING_SQ_NEED_WAKEUP twice, put it inside
sq_ring_needs_enter().
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Works just like io_uring_submit(), but also allows retrieving events
(or waiting/polling for them) in the same call.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_uring_submit() is a last commit point when a caller has to be sure
that all prepared sqes eventually lands to the kernel.
If SQ thread is running it is quite possible that khead != ktail, thus
charging of sq->array is skipped, which leads to IO hang.
Nasty hang is well reproduced when submitter and completion harverster
are different threads.
Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Enter kernel only if SQ thread is off or wakeup is needed.
Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Use proper unsigned math to figure out how many entries we have. If
we have head and tail on either side of UINT_MAX, then we currently
don't submit anything as:
while (head < tail) {
is never true.
Reported-by: Weiping Zhang <zhangweiping@didiglobal.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
We have io_uring_get_sqe() on the submission side, yet the completion
side is named _completion. Rename as follows:
io_uring_get_completion() io_uring_peek_cqe()
iO_uring_wait_completion() io_uring_wait_cqe()
This better tells the user what the _get variant does by calling it
_peek instead, and we move to using _cqe() as the postfix instead
of _completion.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
There's a failure case where an application gets a cqe entry, but
the kernel can then overwrite it before the application is done
reading it. This can happen since the io_uring_{get,wait}_completion()
interface both returns a CQE pointer AND increments the ring index.
If the kernel reuses this entry before the applications is done reading
it, the contents may be corrupted.
Remove the CQ head increment from the CQE retrieval, and put it into
a separate helper, io_uring_cqe_seen(). The application must call this
helper when it got a new CQE entry through one of the above calls, and
it's now done reading it.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
A previous fix that ensured we pass back the right error messed
up the normal return, which is number of entries submitted.
This makes the poll test cases fail.
Fixes: 8260029608b9 ("queue: ensure io_uring_submit() returns the right error")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
We weren't passing back -errno for the system call failure.
This meant any error got turned into EPERM as far as the
caller was concerned.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
This really wants to be a "will we over-fill the ring?" kind of
check, but the sqe_head/sqe_tail should not be that far apart. If they
are, that's a bug elsewhere. So just kill the check.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Applications should not need to care about this, we can pass it in
ourselves. Once the libc support is there, we won't expose this
parameter either.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Update liburing and io_uring_enter.2 to match the kernel.
Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Let's have the various helpers be in usefully named functions, no
need to bundle them all into the same one.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
