UNIX has one (to my mind) big hole in its API. Windows, surprisingly, doesn\u00e2\u20ac\u2122t share it. This article will discuss that gap, and how it can be worked around.<\/p>\n
There are a few multiplexing system calls, As well as monitoring an open file or socket, there are handy ways of making a descriptor that can be waited on in one thread, and triggered in another. For example, UNIX signals can be coaxed into fitting this model with Timers can be created to notify via a file descriptor using We can monitor for filesystem changes with All these combine to mean we can, by use of the multiplexing API, listen simultaneously for any mixture of these events, and handle each appropriately as it happens.<\/p>\n Many of the pthread library functions can put a thread to sleep, the core mutex, for example, will sleep until any other thread holding that lock releases it. In theory, that could be used as a wake mechanism. Better though is to use the system intended for event notification across threads \u00e2\u20ac\u201c the wait condition.<\/p>\n A wait condition is really just a handle that one or more threads reference, and sleep on, and any other thread can trigger a wake.<\/p>\n There is no native API for multiplexing wait conditions. You cannot wait for conditionA<\/em> or conditionB<\/em> in one call.<\/p>\n The hole in UNIX is exemplified by a requirement like this:<\/p>\n Run a thread that wakes up when a network connection is received, or when, say, a new command is pushed onto a command queue<\/em>.<\/p>\n The problem is that there is no way to bridge wait conditions into the multiplexed APIs. You can\u00e2\u20ac\u2122t sleep on a file descriptor or<\/em> wait condition; you can only sleep on one. Worse, as mentioned above, you cannot even wait on multiple wait conditions at once.<\/p>\n The common solution<\/a> is to abandon wait conditions and use This will work but has the distinct drawback of using up a file descriptor \u00e2\u20ac\u201c a limited resource. It also means you lose a lot of flexibility, because you have to centralise that file descriptor so that it\u00e2\u20ac\u2122s known and monitored by the thread that wakes and<\/em> the thread that sleeps. Your design becomes incapable of abstracting the supply of events \u00e2\u20ac\u201c you have to tie the event generator and the event receiver closely together.<\/p>\n This makes it hard to write a library module that supplies events. It ends up having to know who its supplying events to.<\/p>\n My solution is one that I based on an obscure article I found by Sun, which now that Sun has gone, I can\u00e2\u20ac\u2122t find online. It talked about implementing a Windows-style The key observation we need to make is that we really can only wait for one wait condition at a time on UNIX. Therefore we are never going to be able to call a single function that takes multiple handles created by The typical form of use of one of these wait conditions would be something like this:<\/p>\n Then of course, having been led down that path, we create another wait condition for monitoring an event on, say, We have to focus instead on what we want: the ability to wake on any of multiple events. The trick is the realisation that we can represent those multiple events with a single wait condition.<\/p>\n I\u00e2\u20ac\u2122ve left the individual events in too as other threads might monitor them alone, but the important thing to note is that when we want to wait on multiple events we still only wait on one condition. The difference is that it\u00e2\u20ac\u2122s a composite condition, representing all of the possible wake conditions.<\/p>\n This sort of code is obviously unwieldy, we would have to alter every bit of code that modifies a variable to tell it about every bit of code that includes a monitor of that variable \u00e2\u20ac\u201c worse the thread that modifies The solution is to abstract away the multiple broadcasts and multiple locks so that neither side needs to know anything about the other. Clearly then there are two sides: the notifier and the waiter. Here\u00e2\u20ac\u2122s what we\u00e2\u20ac\u2122ll aim for (I\u00e2\u20ac\u2122m going to flip to C++ now as we\u00e2\u20ac\u2122ll get needlessly messy code without some object oriented facilities):<\/p>\n Notice that the It\u00e2\u20ac\u2122s all very well to show the end point, how do we do it though? We invert the control \u00e2\u20ac\u201c instead of having a wait condition per condition, we have a wait condition per waiter, and attach as many events to it as we want.<\/p>\n The following is nowhere near complete, nor even thread-safe, but it captures the core idea.<\/p>\n This is an extraordinarily sparse implementation as I\u00e2\u20ac\u2122m already well passed my quota for code in a blog post, I\u00e2\u20ac\u2122ll get around to a fuller, more general implementation and will post it on my github page eventually (emails to prod me when I forget are welcome).<\/p>\n The important point to note is that these two objects provide, effectively, two table joins (although there is no database of course). One maintains the list of events that can wake it, and the other maintains the list of listeners that should be woken. That is what gives us our strong isolation When a waiter subscribes to an event it does not care nor know if there are fifty others waiting on the same condition. When a supplier of a notification triggers a wake it does not care if one of its waiters is waiting for fifty other events too.<\/p>\n I\u00e2\u20ac\u2122m conscious now that I\u00e2\u20ac\u2122ve already (as is my habit) written too much for a single post. I\u00e2\u20ac\u2122ll return to this next time when I\u00e2\u20ac\u2122ll discuss using this technique to blend wait conditions and file descriptor multiplexing. We\u00e2\u20ac\u2122ll also ditch the UNIX has one (to my mind) big hole in its API. Windows, surprisingly, doesn\u00e2\u20ac\u2122t share it. This article will discuss that gap, and how it can be worked around. File Descriptor Wakes There are a few multiplexing system calls, select(), poll(), and epoll() that put a thread to sleep, with the wakeup triggered by activity\u2026 Read More »<\/a><\/span><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[42,6,81,80],"_links":{"self":[{"href":"https:\/\/www.fussylogic.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1067"}],"collection":[{"href":"https:\/\/www.fussylogic.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fussylogic.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fussylogic.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fussylogic.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1067"}],"version-history":[{"count":6,"href":"https:\/\/www.fussylogic.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1067\/revisions"}],"predecessor-version":[{"id":1205,"href":"https:\/\/www.fussylogic.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1067\/revisions\/1205"}],"wp:attachment":[{"href":"https:\/\/www.fussylogic.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1067"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fussylogic.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1067"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fussylogic.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1067"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}select()<\/code>, poll()<\/code>, and epoll()<\/code> that put a thread to sleep, with the wakeup triggered by activity on particular file descriptors. The modern choice is epoll()<\/code>, but classic examples always use select()<\/code>. They\u00e2\u20ac\u2122re the same at their core.<\/p>\neventfd()<\/code> and pipe()<\/code>.<\/p>\nsignalfd()<\/code>, which removes some of the async signal handling problems that can plague systems software.<\/p>\ntimerfd_xxx()<\/code> family functions.<\/p>\ninotify()<\/code>.<\/p>\nThread Wakes<\/h1>\n
Never The Twain<\/h1>\n
Solution #1<\/h1>\n
eventfd()<\/code> or pipe()<\/code> as the event signalling mechanism.<\/p>\nSolution #2<\/h1>\n
WaitFor<\/code> call<\/a>. The advantage that Windows has in this case is that file descriptors are perfectly acceptable event handles for it\u00e2\u20ac\u2122s WaitFor<\/code> family. However, Sun\u00e2\u20ac\u2122s example was only about implementing the ability to wait on multiple conditions in parallel, not wait conditions and file descriptors. We\u00e2\u20ac\u2122ll discuss just that simple case first, and move on to mixing in file descriptors in a later article.<\/p>\nMultiplexed Wait Conditions<\/h2>\n
pthread_cond_init()<\/code> and wakes on any of them.<\/p>\nvoid<\/span> waitingThread() {\n \/\/ --- waiting thread<\/span>\n pthread_cond_init( &conditionXGreaterThan10, null );\n \/\/ ...<\/span>\n pthread_mutex_lock( &lockX );\n if<\/span>( !(x > 10<\/span>) )\n pthread_cond_wait( &conditionXGreaterThan10, &lockX );\n pthread_mutex_unlock( &lockX );\n}\n\nvoid<\/span> threadThatModifiesX() {\n \/\/ --- thread that modifies X<\/span>\n pthread_mutex_lock( &lockX );\n x = newValueOfX;\n if<\/span>( x > 10<\/span> )\n pthread_cond_broadcast( &conditionXGreaterThan10 );\n pthread_mutex_unlock( &lockX );\n}<\/code><\/pre>\ny == 0<\/code>. That all then collapses when we want one thread to also wake on either condition.<\/p>\nvoid<\/span> waitingThread() {\n pthread_cond_init( &conditionXGreaterThan10, null );\n pthread_cond_init( &conditionYEquals0, null );\n pthread_cond_init( &conditionXorYEvent, null );\n \/\/ ...<\/span>\n \/\/ conditionXGreaterThan10 code as above<\/span>\n \/\/ ...<\/span>\n pthread_mutex_lock( &lockXY );\n if<\/span>( !(x > 10<\/span> || y == 0<\/span>) )\n pthread_cond_wait( &conditionXorY, &lockXY );\n pthread_mutex_unlock( &lockXY );\n}\n\nvoid<\/span> threadThatModifiesX() {\n pthread_mutex_lock( &lockX );\n pthread_mutex_lock( &lockXY );\n x = newValueOfX;\n if<\/span>( x > 10<\/span> ) {\n pthread_cond_broadcast( &conditionXGreaterThan10 );\n pthread_cond_broadcast( &conditionXorYEvent );\n }\n pthread_mutex_unlock( &lockXY );\n pthread_mutex_unlock( &lockX );\n}\n\nvoid<\/span> threadThatModifiesY() {\n \/\/ --- thread that modifies y<\/span>\n pthread_mutex_lock( &lockY );\n pthread_mutex_lock( &lockXY );\n y = newValueOfY;\n if<\/span>( y == 0<\/span> ) {\n pthread_cond_broadcast( &conditionYEquals0 );\n pthread_cond_broadcast( &conditionXorYEvent );\n }\n pthread_mutex_unlock( &lockXY );\n pthread_mutex_unlock( &lockY );\n}<\/code><\/pre>\nx<\/code> has to know at least a little bit about y<\/code> thanks to lockXY<\/code>. Very bad for code isolation.<\/p>\n\/\/ --- global<\/span>\nint<\/span> x = 0<\/span>;\nint<\/span> y = 100<\/span>;\nTNotifier notifyXGreaterThan10;\nTNotifier notifyYEquals0;\n\nvoid<\/span> waitingThread() {\n TEventWaiter EW;\n EW.waitfor( notifyXGreaterThan10 );\n EW.waitfor( notifyYEquals0 );\n EW.sleep();\n\n \/\/ ... interrogate EW for waker ...<\/span>\n}\n\nvoid<\/span> notifyingThread1() {\n \/\/ --- notifying thread #1<\/span>\n notifyXGreaterThan10.lock();\n x = newValueOfX;\n if<\/span>( x > 10<\/span> )\n notifyXGreaterThan10.broadcast();\n notifyXGreaterThan10.unlock();\n}\n\nvoid<\/span> notifyingThread2() {\n \/\/ --- notifying thread #2<\/span>\n notifyYEquals0.lock();\n y = newValueOfX;\n if<\/span>( y > 10<\/span> )\n notifyYEquals0.broadcast();\n notifyYEquals0.unlock();\n}<\/code><\/pre>\nx<\/code>-modifying thread no longer needs to know anything about y<\/code>; notice also that we\u00e2\u20ac\u2122re back to having the original simple appearance that the first single-condition example had.<\/p>\nclass TEventWaiter {\n public:\n TEventWaiter() :\n mEvent(PTHREAD_COND_INITIALIZER),\n mConditionListLock(PTHREAD_MUTEX_INITIALIZER)\n {\n }\n\n ~TEventWaiter() {\n \/\/ ... iterate through condition list and unsubscribe<\/span>\n }\n\n bool sleep() {\n pthread_mutex_lock( &mConditionListLock );\n pthread_cond_wait( &mEvent, &mConditionListLock );\n pthread_mutex_unlock( &mConditionListLock );\n }\n\n void<\/span> waitfor(TNotifier *notifier) {\n pthread_mutex_lock( &mConditionListLock );\n mConditions.push_back(notifier);\n notifier->subscribe(this);\n pthread_mutex_unlock( &mConditionListLock );\n }\n\n void<\/span> broadcast( const<\/span> TNotifier * ) {\n pthread_cond_broadcast( &mEvent );\n }\n\n protected:\n pthread_cond_t mEvent;\n mutable pthread_mutex_t mConditionListLock;\n list<TNotifier*> mConditions;\n};\n\nclass TNofifier {\n public:\n TEventWaiter() :\n mSubscriberListLock(PTHREAD_MUTEX_INITIALIZER)\n {\n }\n\n \/\/ Tell every subscriber the event has occurred<\/span>\n void<\/span> broadcast() const<\/span> {\n pthread_mutex_lock( &mSubscriberListLock );\n list<TEventWaiter*>::const_iterator it;\n for<\/span>( it = mSubscribers.begin(); it != mSubscribers.end(); it++ ) {\n it->broadcast( this );\n }\n pthread_mutex_unlock( &mSubscriberListLock );\n }\n\n \/\/ Allow a waiter to subscribe to us<\/span>\n void<\/span> subscribe(TEventWaiter* subscriber) {\n pthread_mutex_lock( &mSubscriberListLock );\n mSubscribers.push_back(subscriber);\n pthread_mutex_unlock( &mSubscriberListLock );\n }\n\n protected:\n mutable pthread_mutex_t mSubscriberListLock;\n list<TEventWaiter*> mSubscribers;\n};<\/code><\/pre>\npthreads<\/code> dependence, and use the nice new C++11 concurrency libraries instead.<\/p>\n","protected":false},"excerpt":{"rendered":"