US8364862B2

US8364862B2 - Delegating a poll operation to another device

Info

Publication number: US8364862B2
Application number: US12/482,614
Authority: US
Inventors: Michael J. Espig; Zhen Fang; Ravishankar Iyer; David J. Harriman
Original assignee: Intel Corp
Current assignee: Intel Corp
Priority date: 2009-06-11
Filing date: 2009-06-11
Publication date: 2013-01-29
Also published as: CN102804155A; CN102804155B; DE112010002425B4; US20100318693A1; US8762599B2; DE112010002425T5; WO2010144164A1; US20130138843A1

Abstract

In one embodiment, the present invention includes a method for handling a registration message received from a host processor, where the registration message delegates a poll operation with respect to a device from the host processor to another component. Information from the message may be stored in a poll table, and the component may send a read request to poll the device and report a result of the poll to the host processor based on a state of the device. Other embodiments are described and claimed.

Description

BACKGROUND

Modern computer systems typically include a processor and various other components that are coupled together. In addition, many systems include one or more peripheral or input/output (IO) devices.

To enable communications between software that executes on the processor and operations that may be performed by the other devices, different mechanisms can be used. Common mechanisms include a polling method and an interrupt method. However, neither of these methods is optimal. Using a polling technique, software either continuously polls status registers on the IO device if the IO device's task is fine-grained, or relies on an asynchronous interrupt through the operating system (OS) if the IO device's task is coarse-grained. While a polling method may ensure good performance, it suffers from drawbacks. First, the core/thread that needs to know the completion status has to continuously check (e.g., via a busy spin operation) on a memory mapped input/output (MMIO) status register, preventing itself from entering a low power state. Second, repetitive polling on an uncacheable MMIO address results in a large amount of traffic on a system interconnect. In a word, the fast response time comes at a cost of power consumption (a major issue especially for ultra-low power environments) and waste of system resources.

An interrupt method avoids busy spinning of a processor on the status register. While waiting, the core/thread can either context switch to execute another process or enter a lower power state. Completion of the task on the IO device triggers an interrupt into the OS. However, in a typical system, several hundred cache misses and tens of thousand clock cycles are induced by a kernel interrupt handler. This performance overhead of interrupt handling is not acceptable for many fine-grained logic blocks.

Thus both polling and interrupt techniques are not satisfactory for a low power application, as polling negates a large portion of any power benefits from using an IO device, while interrupts introduce a large performance penalty.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system in accordance with one embodiment of the present invention.

FIG. 2 is a flow diagram of a method in accordance with one embodiment of the present invention.

FIG. 3 is a block diagram of a system in accordance with another embodiment of the present invention.

DETAILED DESCRIPTION

In various embodiments, a poll delegation technique may be implemented in which an interconnect serves as a delegate in a polling and notification process. In one embodiment the interconnect may be an input/output (IO) interconnect, although the scope of the present invention is not limited in this regard. Using this technique, the interconnect polls IO devices for a host processor such as a central processing unit (CPU) and notifies an application software of a given event using one of a number of techniques such as a test and hold operation, or by update to a user-selected memory location that triggers a processor's exit from a power optimized state. In one embodiment, user-level instructions such as MONITOR/MWAIT may be used to notify application software. In various embodiments, poll delegation may enable a response time as short as polling, and power consumption/resource usage as low as an interrupt-based technique, thus providing user-level notification of IO device status without the need for polling or interrupts.

In one embodiment the IO interconnect may include specific-purpose hardware to poll status register(s) of an IO device. Then upon a status change, the IO interconnect can issue a write operation (e.g., a coherent write) to the memory address that is being monitored by the host. The coherent write will be detected by this hardware, and cause the thread that is waiting on the address to resume execution. Thus in various embodiments, a processor can stay in a low power state until the IO device is done with its task, and resume execution almost as fast as if it had been busy-spinning. No change is required in the processor core, the cache, the system coherent interconnect, or the IO devices.

The MONITOR/MWAIT pair of instructions can support inter-thread synchronization. The instruction pair can be available at all privilege levels. MONITOR can be used to enable a CPU to set up monitoring hardware of the CPU to detect stores to an effective address range (typically a cacheline). This address range belongs to a coherent, write-back address range. In one embodiment, cache coherency hardware may monitor for a write to the destination address. When that write occurs the cache coherency controller will send a message to the processor to come out of the low power state. After this set up, the succeeding MWAIT instruction puts the processor core into a selected low-power state (e.g., a clock-gated state or a power-gated state). When the monitoring hardware detects a store to any byte in the address range, the stalled thread resumes execution from the instruction following MWAIT. Architecturally, MWAIT behaves like a no operation (NOP). While these MONITOR and MWAIT instructions are designed to implement performance and power-optimized inter-thread synchronization, embodiments can leverage the instructions for IO device completion notification.

Referring now to FIG. 1, shown is a block diagram of a system in accordance with one embodiment of the present invention. As shown in FIG. 1, system 100 may be a system on a chip (SoC) that includes various components interconnected together and which may provide mechanisms to handle polling delegation in accordance with an embodiment of the present invention. Specifically, in the embodiment of FIG. 1, SoC 100 may include a plurality of processor cores only one of which, core 110 _a, is shown for ease of illustration.

The one or more cores may be coupled via a coherent interconnect 115 to one or more cache memory 120 _a. Coherent interconnect 115 may include various hardware, software and/or firmware to implement a cache coherency protocol, such as a modified exclusive shared invalid (MESI) protocol, to maintain a coherent view of information stored within the system. In some implementations, coherent interconnect 115 may be a layered protocol including various layers such as a protocol layer, a link layer and possibly a physical layer (where the system is not on a single die).

In turn, coherent interconnect 115 may be coupled via a hub 120 to a memory controller 130 that in turn may be coupled to a system memory, e.g., dynamic random access memory (DRAM), for example. Note that such memory is not shown in FIG. 1, as the memory may be external to the SoC.

In addition, coherent interconnect 115 may be coupled to an upstream side of an IO interconnect 140 which may be of a given communication protocol such as a Peripheral Component Interconnect Express (PCI Express™ (PCIe™)) protocol in accordance with links based on the PCI Express™ Specification Base Specification version 2.0 (published Jan. 17, 2007) (hereafter the PCIe™ Specification), or another such protocol. IO interconnect 140 may include a polling table 150 in accordance with an embodiment of the present invention. While shown as being present in the interconnect, other implementations may locate this buffer elsewhere in close relation to the interconnect. In turn, various devices, e.g., devices 160 ₀and 160 ₁, which may be IO devices, intellectual property (IP) blocks or so forth may be coupled to a downstream side of IO interconnect 140.

As seen in FIG. 1, polling table 150 stores a plurality of entries including, e.g., entry 156. Each entry may include a device monitored location such as an address of a status register present in one of devices 160, a memory monitored address, which may correspond to a physical address in system memory that corresponds to a monitored location for an MWAIT instruction, and a initial value, which may be the initial value of the device monitored location, e.g., the initial value of the status register. Embodiments thus essentially associate each status register on an IO device with a corresponding memory address that is being monitored by the CPU. While shown with this particular implementation in the embodiment of FIG. 1, the scope of the present invention is not limited in this regard.

Referring now to FIG. 2, shown is a flow diagram of a method in accordance with one embodiment of the present invention. Specifically, FIG. 2 shows a flow diagram that may be performed by logic of an IO or other interconnect that handles poll delegation on behalf of a host processor. In some implementations, method 200 may be implemented by way of a state machine or other logic of an interconnect, although other implementations are possible. As shown in FIG. 2, method 200 may begin by receiving a registration message from an application (block 210). In one embodiment, the registration message may include a tuple of MMIO address, true physical memory address, and initial value. In one embodiment, this registration message may include a physical address range for a location and a monitored system memory that backs up a monitored virtual address range, a MMIO address for obtaining the status of a location in a device to be monitored, e.g., a status register, and an initial value of such register, which may also correspond to the initial value stored in the monitored address of system memory. In one embodiment, an application may make a system called to pass this information to the interconnect, e.g., using a host processor.

Still referring to FIG. 2, it may be determined whether an entry is available in the polling table of the interconnect (diamond 215). If not, a failure message may be sent back to the host (block 220). Otherwise, the information may be stored into an entry of the polling table (block 230). This information thus may correspond to a device monitored location (i.e., corresponding to a MMIO address of a completion status register of the device), an initial value of the register, and a memory monitored address (which may be a physical address location within the system memory that is monitored, e.g., by a MONITOR/MWAIT instruction pair). Accordingly, a success message may be sent from the interconnect to the host processor (block 240).

During operation, the application then initializes the monitored location, issuing the MONITOR and MWAIT instructions, thus enabling the device to begin executing its task. Various such tasks may be realized, including offloading of specialized functions, graphics processing, physics processing or so forth. As one example, the function may be a specialized calculation such as a fast fourier transform (FFT). The application thus may pass various information regarding the FFT such as the number of points, the starting address and so forth, prior to execution of the MONITOR/MWAIT instructions.

Accordingly, at this time one or more cores of the host processor may enter a low power state, which may be configurable depending on a type of operation that the device is to perform.

Referring still to FIG. 2, after the device begins executing its operation, the interconnect may perform polling at the device monitored locations (block 245). In one embodiment, the polling operation may issue a read operation to each register address in the polling table, and compare the fetched value against the initial value in the polling table entry, as discussed with regard to diamond 250. Understand that such polling may poll a number of locations present in one or more devices, namely whatever addresses are indicated in a polling table. It may be determined on each polling operation whether there is a change in any entries' value from its initial value (diamond 250). If not, a further polling iteration may occur. If a change occurs, control instead passes to block 260, where a write operation may be issued to the memory monitored address with the new value. In one embodiment, the interconnect may issue a coherent write to this address, which may be realized using a message signaling interrupt (MSI), in one embodiment. This coherent write to the monitored address in memory will cause the processor core to wake up from the low power state, enabling the halted thread to proceed. With reference to FIG. 2, it may be determined after issuing the write operation whether a release of the entry has been received (diamond 270). In one embodiment, such a release may be received from the OS when a given application associated with the poll delegation entry terminates. Accordingly, if the device knows the task is done, the device could issue an eviction request to the IO interconnect. Control passes from diamond 270, if the release is not indicated, back to block 245, where further polling may be performed. While shown with this particular implementation in the embodiment of FIG. 2, the scope of the present invention is not limited in this regard.

The generation of the registration message and release message may use help from the OS. The user-level MONITOR and MWAIT instructions may be performed completely in user mode, and the poll delegation operation can be purely hardware. Note that the registration and release are usually only performed at the application initialization and cleanup phases. Therefore their power and performance do not matter. In contrast, the user-level setup and poll detection usually are executed a large number of times. The efficiency of these two steps thus enables efficient system power and performance characteristics.

To support multiple IP accelerators, polling table 150 may be a multi-entry translation table. In one embodiment, each entry 156 contains an MMIO address and the physical memory address that is linked to it. In some implementations, the number of entries in the polling table can be a small number, e.g., N=8. In the extreme case when more than N registers need to be checked at the same time, the user thread can always directly poll the registers instead of using the above method, although a virtualized polling table could instead be used.

In one embodiment, a message signaling interrupt (MSI-X) feature of PCI that may be in IO interconnect 140 provides hardware that allows devices 160 to issue writes to system memory locations. In MSI-X, the target memory locations are special addresses that will lead to interrupts. In embodiments instead addresses in write back memory space can be used as targets of such writes. The property of the target memory location is transparent to MSI-X hardware, it simply delivers a packet from the IO interconnect to the memory system. In some implementations the polling performed by interconnect 140 may cause a poll of registered status register addresses, even if the devices that some of the registers represent not actively computing. This is because in some implementations the poll delegation logic 145 has no knowledge of whether or not a valid entry in its mapping table represents an inactive device. Different mechanisms can be used to provide this information to the logic. For example, a system call can be provided by the OS to allow an application to release a particular entry in the polling table. Alternatively, the IO interconnect 140 could intercept power-state transition commands that are sent to the IP blocks 160 so that it will know which status register will not be updated any time soon. This information may be included in a status portion of the entries of polling table 150 in such embodiments. It is noted however, that the cost for the interconnect 140 to poll IO registers is rather low, and as such the power and performance impact of indiscriminate polling may be minimal.

While described herein for a system-on-chip (SoC) configuration, which may be the primary processing component for a computing device such as an embedded, portable or mobile device, other implementations may be used in other systems such as multiprocessor computer systems having a processor coupled to a coherent interconnect, that in turn may be coupled to an IO interconnect via one or more chipsets or other components. Still further, embodiments may be implemented in a multi-chip architecture for a computing device.

Referring now to FIG. 3, shown is a block diagram of a system in accordance with another embodiment of the present invention. As shown in FIG. 3, system 300 may be multi-chip architecture, namely a system 300 including a first chip 100, which may be a first SoC, and which may be configured the same as that of FIG. 1, a second integrated circuit 310, which may provide peripheral functionality, and a memory 370, which may be a DRAM coupled to one or more of the chips. As shown in FIG. 3, SoC 100 may communicate with IC 310 via an interconnect 305 that in turn is coupled to a first bridge 170 of SoC 100 and a second bridge 320 of IC 310. In turn, second bridge 320 may be coupled to an IO interconnect 330 that in turn can be coupled to one or more peripheral devices, e.g., a PCIe device 340 and a universal serial bus (USB) device 350. Still further, IO interconnect 330 may be coupled via a third bridge 360 to off-chip components via a serial over network interface (SONIC)/external fabric. In the embodiment of FIG. 3, note that both I0 interconnect 140 and IO interconnect 330 may include polling tables and poll delegation logic in accordance with an embodiment of the present invention. While shown with this particular implementation in the embodiment of FIG. 3, other implementations are of course possible.

Using an embodiment of the present invention, a process can avoid either suffering from long latency for interrupt handling, or have to busy-spin in a high power state. On a processor that supports MONITOR/MWAIT or similar test and set functions, poll delegation allows the processor to enter power and performance-optimized states while still achieving the quick response time of busy spins. For low-power SoCs that include finer-grained IP blocks, embodiments provide a near-optimal completion notification solution in terms of power and performance.

Embodiments may be implemented in code and may be stored on a storage medium having stored thereon instructions which can be used to program a system to perform the instructions. The storage medium may include, but is not limited to, any type of disk including floppy disks, optical disks, compact disk read-only memories (CD-ROMs), compact disk rewritables (CD-RWs), and magneto-optical disks, semiconductor devices such as read-only memories (ROMs), random access memories (RAMs) such as dynamic random access memories (DRAMs), static random access memories (SRAMs), erasable programmable read-only memories (EPROMs), flash memories, electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, or any other type of media suitable for storing electronic instructions.

While the present invention has been described with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover all such modifications and variations as fall within the true spirit and scope of this present invention.

Claims

1. An apparatus comprising:

a core to generate a registration message to delegate a poll operation to an input/output (IO) interconnect;

a coherent interconnect coupled to the core;

the IO interconnect coupled to the coherent interconnect, the IO interconnect including a poll table having a plurality of entries each having a register address field to store a register address received in a registration message, a destination address field to store a destination address in a system memory received in the registration message, and an initial value field to store an initial value associated with the register address received in the registration message; and

at least one device coupled to the IO interconnect to perform an operation for an application executing on the core and including at least one status register, the IO interconnect to poll the at least one status register responsive to information in a poll table entry, and to issue a write transaction to the destination address if a polled value of the at least one status register differs from the initial value, the IO interconnect further including a poll delegation logic to issue a read request to the at least one device at a predetermined interval to perform the poll.

2. The apparatus of claim 1, wherein the at least one device is to update the initial value to the polled value upon completion of the operation.

3. The apparatus of claim 1, wherein the apparatus comprises a system on a chip (SoC) formed on a single semiconductor die, and the at least one device comprises an intellectual property (IP) block.

4. The apparatus of claim 1, wherein the poll delegation logic is to perform a comparison between data received from the at least one device responsive to the read request and the initial value and to perform the write transaction when the data and the initial value differ.

5. The apparatus of claim 1, wherein the core is to send an eviction message to the IO interconnect to delete an entry in the poll table when an application corresponding to the entry is terminated.

6. The apparatus of claim 1, wherein the core is to execute a first instruction to set up the destination address and a second instruction to cause the core to enter a low power state until the destination address is updated.

7. The apparatus of claim 6, wherein the core is to send the registration message to the IO interconnect prior to execution of the first and second instructions.

8. A system comprising:

a first integrated circuit including:

at least one core including a first logic to execute a first instruction to set up a monitored address in a memory and a second logic to cause the at least one core to enter a low power state when a predetermined instruction follows the first instruction;

a first coherent interconnect coupled to the at least one core;

a first input/output (IO) interconnect including a poll table to store a tuple including a register identifier of a register in an intellectual property (IP) block coupled to the first IO interconnect, the monitored address in the memory, and an initial value associated with the register, and a delegation logic to receive a delegation message from the at least one core, and based on the tuple, obtain a current value of the register until the current value differs from the initial value and responsive to the difference write data to the monitored address; and

the IP block coupled to the first IO interconnect including the register and to perform a function for an application executing on the at least one core; and

the memory coupled to the first integrated circuit via a memory interconnect, wherein the at least one core is to exit the low power state responsive to the data being written to the destination address of the memory and to continue execution of the application at a next instruction following the predetermined instruction.

9. The system of claim 8, wherein the IP block is to update the initial value to the current value upon completion of the operation.

10. The system of claim 8, wherein the at least one core is to send the delegation message to the first IO interconnect with the tuple prior to execution of the first instruction.

11. A method comprising:

receiving a registration message from a host processor in an interconnect coupled between the host processor and a device, the registration message to delegate a poll operation with respect to the device to the interconnect;

storing information regarding a device monitored location, a memory monitored address, and an initial value of the device monitored location in a poll table associated with the interconnect; and

using a poll delegation logic of the interconnect to send a read request from the interconnect to the device to poll the device and to compare the initial value with a device value obtained from the device monitored location, and reporting a result of the poll to the host processor if the device value is different than the initial value.

12. The method of claim 11, further comprising issuing a write request from the interconnect to the memory monitored address in a system memory to report the result.

13. The method of claim 12, further comprising issuing a message signaling interrupt from the interconnect to the system memory to issue the write request.

14. The method of claim 13, wherein the host processor is placed into a low power state after sending the registration message, and responsive to the write to the memory monitored address, the host processor is to exit the low power state.

15. The method of claim 11, wherein the poll table includes a plurality of entries each including a device monitored location, a memory monitored address, and an initial value, and further comprising sending a read request to the device for each of the plurality of entries.

16. The method of claim 11, wherein the device monitored location corresponds to a status register of the device, and associating the status register with the memory monitored address via the poll table.

17. The method of claim 11, further comprising initiating the registration message via an application, and performing a first function on the device for the application, wherein the application is to initiate the first function after the information is stored in the poll table, and to cause the host processor to enter into a low power state.