This paper describes the motivation, design and performance of Midway, a programming system for a distributed shared memory multicomputer (DSM) such an ATM based cluster, a CM-5, or a Paragon. Midway supports a new memory consistency model called entry consistency. Entry consistency guarantees that shared data becomes consistent at a processor when the processor acquires a synchronization object known to guard the data. Entry consistency is weaker than other models described in the literature, such as processor consistency and release consistency, but it makes possible higher performance implementations of the underlying consistency protocols. Midway programs are written in C, and the association between the synchronization objects and data must be made with explicit annotations. As a result, pure entry consistent programs can require annotations than programs written to other models. In addition to entry consistency, Midway also supports the stronger release consistent and processor consistent models at the granularity of individual data items. Consequently, the programmer can tradeoff potentially reduced performance for the additional programming complexity required to write an entry consistent parallel program.