US20020073360A1 - Methods, systems, and articles of manufacture for analyzing performance of application programs - Google Patents
Methods, systems, and articles of manufacture for analyzing performance of application programs Download PDFInfo
- Publication number
- US20020073360A1 US20020073360A1 US09/244,895 US24489599A US2002073360A1 US 20020073360 A1 US20020073360 A1 US 20020073360A1 US 24489599 A US24489599 A US 24489599A US 2002073360 A1 US2002073360 A1 US 2002073360A1
- Authority
- US
- United States
- Prior art keywords
- performance
- data
- program
- executing
- thread
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3604—Software analysis for verifying properties of programs
- G06F11/3612—Software analysis for verifying properties of programs by runtime analysis
Definitions
- the present invention relates generally to performance analysis and more specifically to methods for providing a multi-dimensional view of performance data associated with an application program.
- Multi-threading is the partitioning of an application program into logically independent “threads” of control that can execute in parallel.
- Each thread includes a sequence of instructions and data used by the instructions to carry out a particular program task, such as a computation or input/output function.
- each processor executes one or more threads depending upon the number of processors to achieve multi-processing of the program.
- a program can be multi-threaded and still not achieve multi-processing if a single processor is used to execute all threads. While a single processor can execute instructions of only one thread at a time, the processor can execute multiple threads in parallel by, for example, executing instructions corresponding to one thread until reaching a selected instruction, suspending execution of that thread, and executing instructions corresponding to another thread, until all threads have completed. In this scheme, as long as the processor has started executing instructions for more than one thread during a given time interval all executing threads are said to be “running” during that time interval.
- Multiprocessor computer systems are typically used for executing application programs intended to address complex computational problems in which different aspects of a problem can be solved using portions of a program executing in parallel on different processors.
- a goal associated with using such systems to execute programs is to achieve a high level of performance, in particular, a level of performance that reduces the waste of the computing resources.
- Computer resources may be wasted, for example, if processors are idle (i.e., not executing a program instruction) for any length of time.
- Such a wait cycle may be the result of one processor executing an instruction that requires the result of a set of instructions being executed by another processor.
- Performance analysis in this regard generally requires gathering information in three areas.
- a processor's state refers to the portion of a program (for example, set of instructions such as a subprogram, loop, or other code block) that the processor is executing during a particular time interval.
- These three areas do not provide a complete analysis, however. They fail to address a fourth component of performance analysis, namely, precisely what a processor did during a particular state (e.g., computation, input data, output data, etc.).
- a performance analysis tool can determine the affect of operations within a state on the performance level. Once these factors are identified, it is possible to synchronize operations that have a significant impact on performance with operations that have a less significant impact, and achieve a better overall performance level. For example, a first thread may perform an operation that uses significant resources while another thread scheduled to perform a separate operation in parallel with the first thread sits idle until the first thread completes its operation. It may be desirable to cause the second thread to perform a different operation that does not require the first thread to complete its operation, thus eliminating the idle period for the second thread. By changing the second thread's schedule in this way the operations performed by both threads are better synchronized.
- a performance analysis tool When a performance analysis tool reports a problem occurring in a particular state, but fails to relate the problem to other events occurring in an application (for example, operations of another state), the information reported is relatively meaningless.
- a performance analysis tool must assist a developer in determining how performance information relates to a program's execution. Therefore, allowing a developer to determine the context in which a performance problem occurs, provides insight into diagnosing the problem.
- Instrumentation generally requires adding instructions to a program under examination so that when the program is executed the instructions generate data from which the performance information can be derived.
- a subprogram level instrumentation method of performance analysis tracks the number of subprogram calls by instrumenting each subprogram with a set of instructions that generate data reflecting calls to the subprogram. It does not allow a developer to track performance data associated with the operations performed by each subprogram or a specified portion of the subprogram, for example, by specifying data collection beginning and ending points within a subprogram.
- a bucket level instrumentation performance analysis tool divides the executable code into evenly spaced groups, or buckets. Performance data tracks the number of times a program counter was in a particular bucket at the conclusion of a specified time interval. This method of gathering performance data essentially takes a snapshot of the program counter at the specified time interval. This method fails to provide comprehensive performance information because it only collects data related to a particular bucket during the specified time interval.
- the current performance analysis methods fail to provide customized collection or output of performance data.
- performance tools only collect a pre-specified set of data to display to a developer.
- Methods, systems, and articles of manufacture consistent with the present invention overcome the shortcomings of the prior art by facilitating performance analysis of multi-threaded programs executing in a data processing system.
- Such methods, systems, and articles of manufacture analyze performance of threads executing in a data processing system by receiving data reflecting a state of each thread executing during a measurement period, and displaying a performance level corresponding to the state of each thread during the measurement period.
- FIG. 1 depicts a data processing system suitable for implementing a performance analysis system consistent with the present invention
- FIG. 2 depicts a block diagram of a performance analysis system operating in accordance with methods, systems, and articles of manufacture consistent with the present invention
- FIG. 3 depicts a flow chart illustrating operations performed by a performance analysis system consistent with an implementation of the present invention.
- FIG. 4 depicts a multi-dimensional display of the performance data associated with an application program that has been instrumented in accordance with an implementation of the present invention.
- Methods, systems, and articles of manufacture consistent with the present invention utilize performance data collected during execution of an application program to illustrate graphically for the developer performance data associated with the program.
- the program is instrumented to generate the performance data during execution.
- Each program thread performs one or more operations, each operation reflecting a different state of the thread.
- the performance data may reflect an overall performance for each thread as well as a performance level for each state within a thread during execution.
- the developer can specify the type and extent of performance data to be collected. By providing a graphical display of the performance of all threads together, the developer can see where to make any appropriate adjustments to improve overall performance by better synchronizing operations among the threads.
- a performance analysis database access language is used to instrument the program in a manner consistent with the principles of the present invention. Instrumentation can be done automatically using known techniques that add instructions to programs at specific locations within the programs, or manually by a developer.
- the instructions may specify collection of performance data from multiple system components, for example, performance data may be collected from both hardware and the operating system.
- a four-dimensional display of performance data includes information on threads, times, states, and performance level.
- a performance analyzer also evaluates quantitative expressions corresponding to performance metrics specified by a developer, and displays the computed value.
- FIG. 1 depicts an exemplary data processing system 100 suitable for practicing methods and systems consistent with the present invention.
- Data processing system 100 includes a computer system 105 connected to a network 190 , such as a Local Area Network, Wide Area Network, or the Internet.
- network 190 such as a Local Area Network, Wide Area Network, or the Internet.
- Computer system 105 contains a main memory 130 , a secondary storage device 140 , a processor 150 , an input device 170 , and a video display 160 . These internal components exchange information with one another via a system bus 165 .
- the components are standard in most computer systems suitable for use with practicing methods and configuring systems consistent with the present invention.
- One such computer system is the SPARCstation from Sun Microsystems, Inc.
- computer system 100 contains a single processor, it will be apparent to those skilled in the art that methods consistent with the present invention operate equally as well with a multi-processor environment.
- Memory 130 includes a program 110 and a performance analyzer 115 .
- Program 110 is a multi-threaded program.
- the program is instrumented with appropriate instructions of the developer's choosing to generate certain performance data.
- Performance analyzer 115 is comprised of two components.
- the first component 115 a is a library of functions to be performed in a manner specified by the instrumented program.
- the second component 115 b is a developer interface that is used for two functions: (1) automatically instrumenting a program; and (2) viewing performance information collected when an instrumented program is executed.
- instrumentation can be done automatically with the use of performance analyzer interface 115 b .
- the developer simply specifies for the analyzer the type of performance data to be collected and the analyzer adds the appropriate commands from the performance analysis database access language to the program in the appropriate places.
- Techniques for automatic instrumentation in this manner are familiar to those skilled in the art.
- the developer may manually insert commands from the performance analysis database access language in the appropriate places in the program so that during execution specific performance data is recorded.
- the performance data generated during execution of program 110 is recorded in memory, for example, main memory 130 .
- Performance analyzer interface 115 b permits developers to view performance information corresponding to the performance data recorded when program 110 is executed. As explained below, the developer may interact with the analyzer to alter the view to display performance information in various configurations to observe different aspects of the program's performance without having to repeatedly execute the program to collect information for each view, provided the program was properly instrumented at the outset. Each view may show (i) a complete measurement cycle for one or more threads; (ii) when each thread enters and leaves each state; and (iii) selected performance criteria corresponding to each state.
- system 105 has an operating system that controls its operations, including the execution of program 110 by processor 150 .
- main memory 120 main memory
- all or part of systems and methods consistent with the present invention may be stored on or read from other computer-readable media, such as secondary storage devices, like hard disks, floppy disks, and CD-ROM; a carrier wave received from the Internet; or other forms of ROM or RAM.
- secondary storage devices like hard disks, floppy disks, and CD-ROM
- carrier wave received from the Internet or other forms of ROM or RAM.
- data processing system 100 may contain additional or different components.
- FIG. 2 depicts a block diagram of a performance analysis system consistent with the present invention.
- program 210 consists of multiple threads 212 , 214 , 216 , and 218 .
- Processor 220 executes threads 212 , 214 , 216 , and 218 in parallel.
- Memory 240 represents a shared memory that may be accessed by all executing threads.
- a protocol for coordinating access to a shared memory is described in U.S. patent application Ser. No. ______, of Shaun Dennie, entitled “Protocol for Coordinating the Distribution of Shared Memory”, (Attorney Docket No. 06502-0207-00000), which is incorporated herein by reference.
- a single processor 220 is shown, multiple processors may be used to execute threads 212 , 214 , 216 , and 218 .
- an operating system partitions memory 240 into segments designated for operations associated with each thread and initializes the field of each segment.
- memory segment 245 is comprised of enter and exit state identifiers, developer specified information, and thread identification information.
- An enter state identifier stores data corresponding to when, during execution, a thread enters a particular state.
- an exit state identifier stores data corresponding to when, during execution of an application program, a thread leaves a particular state. Developer specified data represents the performance analysis data collected.
- a reserved area of memory 250 is used to perform administrative memory management functions, such as, coordinating the distribution of shared memory to competing threads.
- the reserved area of memory 250 is also used for assigning identification information to threads using memory.
- FIG. 3 provides additional details regarding the operation of a performance analysis system consistent with an implementation of the present invention.
- Instructions that generate performance data are inserted into a program (step 305 ).
- the instrumented program is executed and the performance data are generated (steps 310 and 315 ).
- performance analyzer accesses and displays the performance data (step 320 ).
- Performance analyzer is capable of displaying both the performance data and the related source code and assembly code, i.e., machine instructions, corresponding to the data. This allows a developer to relate performance data to both the source code and the assembly code that produced the data.
- FIG. 4 shows a display 400 with two parts labeled A and B, respectively.
- the first part, labeled A shows the performance characteristics of an application program in four dimensions: threads, time, states, and performance. Performance information for each thread is displayed horizontally using a bar graph-type format. Time is represented on the horizontal axis; performance is represented on the vertical axis.
- Thread 1 and thread 2 in display 400 were executing concurrently. As shown, the threads began executing at different times.
- the horizontal axis for thread 1 is labeled 402 .
- Thread 1 began executing at a point in time labeled “x” on the horizontal axis 402 .
- the horizontal axis for thread 2 is labeled 404 .
- Thread 2 began executing at time “b”.
- Each thread performed operations in multiple states, each state being represented by a different pattern. Thread 2 was idle at the beginning of the measuring period. One reason for this idle period may be that thread 2 was waiting for resources from thread 1 . Based on this information, a developer can allocate operations of a thread among states such that performance will be improved, for example, by not executing concurrent operations that require use of the same system resources.
- thread 1 entered state 410 at a point in time “x” on the horizontal axis 402 and left state 410 at time “y”, and entered state 420 at time “m” and left state 420 at time “n”.
- the horizontal distance between points “x” and “y” is shorter than the horizontal distance between points “m” and “n”. Therefore, thread 1 operated in state 420 longer than it operated in state 410 .
- the vertical height of the bars show a level of performance.
- the vertical height for state 410 is lower than the vertical height for state 420 , showing that states 410 and 420 operated at different levels of performance.
- the change in vertical height as an executing thread transitions from one state to another corresponds to changes in performance level. This information may be used to identify the affect of transitioning between consecutive states on performance, and directs a developer to areas of the program for making changes to increase performance.
- the bottom-half of the display illustrates an expression evaluation feature of the performance analyzer's interface.
- a developer specifies computational to expressions related to a performance metric of a selected state(s).
- the performance analyzer computes the value of an expression for the performance data collected.
- the developer has selected state 440 .
- the expression on the first line, “NUM_OPS/(100000*TIME)”, is an expression for computing the number (in millions) of floating point instructions per second (MFLOPS).
- the expression on the second line, “ 2*_CPU_MHZ” calculates a theoretical peak level of performance for a specified state.
- Performance analyzer may evaluate these two expressions in conjunction to provide quantitative information about a particular state. For example, by dividing MFLOPS by the theoretical peak performance level for state 440, performance analyzer calculates for the developer the percentage of theoretical peak represented by each operation in state 440.
- Methods and systems consistent with the present invention collect performance data from hardware and software components of an application program, allowing a developer to understand how performance data relates to each thread of a program and complementing a developer's ability to understand and subsequently diagnose performance issues occurring in a program.
Abstract
Description
- The present invention relates generally to performance analysis and more specifically to methods for providing a multi-dimensional view of performance data associated with an application program.
- Multi-threading is the partitioning of an application program into logically independent “threads” of control that can execute in parallel. Each thread includes a sequence of instructions and data used by the instructions to carry out a particular program task, such as a computation or input/output function. When employing a data processing system with multiple processors, i.e., a multiprocessor computer system, each processor executes one or more threads depending upon the number of processors to achieve multi-processing of the program.
- A program can be multi-threaded and still not achieve multi-processing if a single processor is used to execute all threads. While a single processor can execute instructions of only one thread at a time, the processor can execute multiple threads in parallel by, for example, executing instructions corresponding to one thread until reaching a selected instruction, suspending execution of that thread, and executing instructions corresponding to another thread, until all threads have completed. In this scheme, as long as the processor has started executing instructions for more than one thread during a given time interval all executing threads are said to be “running” during that time interval.
- Multiprocessor computer systems are typically used for executing application programs intended to address complex computational problems in which different aspects of a problem can be solved using portions of a program executing in parallel on different processors. A goal associated with using such systems to execute programs is to achieve a high level of performance, in particular, a level of performance that reduces the waste of the computing resources. Computer resources may be wasted, for example, if processors are idle (i.e., not executing a program instruction) for any length of time. Such a wait cycle may be the result of one processor executing an instruction that requires the result of a set of instructions being executed by another processor.
- It is thus necessary to analyze performance of programs executing on such data processing systems to determine whether optimal performance is being achieved. If not, areas for improvement should be identified.
- Performance analysis in this regard generally requires gathering information in three areas. The first considers the processor's state at a given time during program execution. A processor's state refers to the portion of a program (for example, set of instructions such as a subprogram, loop, or other code block) that the processor is executing during a particular time interval. The second considers how much time a processor spends in transition from one state to another. The third considers how close a processor is to executing at its peak performance. These three areas do not provide a complete analysis, however. They fail to address a fourth component of performance analysis, namely, precisely what a processor did during a particular state (e.g., computation, input data, output data, etc.).
- When considering what a processor did while in a particular state, a performance analysis tool can determine the affect of operations within a state on the performance level. Once these factors are identified, it is possible to synchronize operations that have a significant impact on performance with operations that have a less significant impact, and achieve a better overall performance level. For example, a first thread may perform an operation that uses significant resources while another thread scheduled to perform a separate operation in parallel with the first thread sits idle until the first thread completes its operation. It may be desirable to cause the second thread to perform a different operation that does not require the first thread to complete its operation, thus eliminating the idle period for the second thread. By changing the second thread's schedule in this way the operations performed by both threads are better synchronized.
- When a performance analysis tool reports a problem occurring in a particular state, but fails to relate the problem to other events occurring in an application (for example, operations of another state), the information reported is relatively meaningless. To be useful a performance analysis tool must assist a developer in determining how performance information relates to a program's execution. Therefore, allowing a developer to determine the context in which a performance problem occurs, provides insight into diagnosing the problem.
- The process of gathering this information for performance analysis is referred to as “instrumentation.” Instrumentation generally requires adding instructions to a program under examination so that when the program is executed the instructions generate data from which the performance information can be derived.
- Current performance analysis tools gather data in one of two ways: subprogram level instrumentation and bucket level instrumentation. A subprogram level instrumentation method of performance analysis tracks the number of subprogram calls by instrumenting each subprogram with a set of instructions that generate data reflecting calls to the subprogram. It does not allow a developer to track performance data associated with the operations performed by each subprogram or a specified portion of the subprogram, for example, by specifying data collection beginning and ending points within a subprogram.
- A bucket level instrumentation performance analysis tool divides the executable code into evenly spaced groups, or buckets. Performance data tracks the number of times a program counter was in a particular bucket at the conclusion of a specified time interval. This method of gathering performance data essentially takes a snapshot of the program counter at the specified time interval. This method fails to provide comprehensive performance information because it only collects data related to a particular bucket during the specified time interval.
- The current performance analysis methods fail to provide customized collection or output of performance data. Generally, performance tools only collect a pre-specified set of data to display to a developer.
- Methods, systems, and articles of manufacture consistent with the present invention overcome the shortcomings of the prior art by facilitating performance analysis of multi-threaded programs executing in a data processing system. Such methods, systems, and articles of manufacture analyze performance of threads executing in a data processing system by receiving data reflecting a state of each thread executing during a measurement period, and displaying a performance level corresponding to the state of each thread during the measurement period.
- The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate an implementation of the invention and, together with the description, serve to explain the advantages and principles of the invention. In the drawings,
- FIG. 1 depicts a data processing system suitable for implementing a performance analysis system consistent with the present invention;
- FIG. 2 depicts a block diagram of a performance analysis system operating in accordance with methods, systems, and articles of manufacture consistent with the present invention;
- FIG. 3 depicts a flow chart illustrating operations performed by a performance analysis system consistent with an implementation of the present invention; and
- FIG. 4 depicts a multi-dimensional display of the performance data associated with an application program that has been instrumented in accordance with an implementation of the present invention.
- Reference will now be made in detail to an implementation consistent with the present invention as illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings and the following description to refer to the same or like parts.
- Overview
- Methods, systems, and articles of manufacture consistent with the present invention utilize performance data collected during execution of an application program to illustrate graphically for the developer performance data associated with the program. The program is instrumented to generate the performance data during execution. Each program thread performs one or more operations, each operation reflecting a different state of the thread. The performance data may reflect an overall performance for each thread as well as a performance level for each state within a thread during execution. The developer can specify the type and extent of performance data to be collected. By providing a graphical display of the performance of all threads together, the developer can see where to make any appropriate adjustments to improve overall performance by better synchronizing operations among the threads.
- A performance analysis database access language is used to instrument the program in a manner consistent with the principles of the present invention. Instrumentation can be done automatically using known techniques that add instructions to programs at specific locations within the programs, or manually by a developer. The instructions may specify collection of performance data from multiple system components, for example, performance data may be collected from both hardware and the operating system.
- A four-dimensional display of performance data includes information on threads, times, states, and performance level. A performance analyzer also evaluates quantitative expressions corresponding to performance metrics specified by a developer, and displays the computed value.
- Performance Analysis System
- FIG. 1 depicts an exemplary
data processing system 100 suitable for practicing methods and systems consistent with the present invention.Data processing system 100 includes acomputer system 105 connected to a network 190, such as a Local Area Network, Wide Area Network, or the Internet. -
Computer system 105 contains amain memory 130, asecondary storage device 140, aprocessor 150, aninput device 170, and avideo display 160. These internal components exchange information with one another via asystem bus 165. The components are standard in most computer systems suitable for use with practicing methods and configuring systems consistent with the present invention. One such computer system is the SPARCstation from Sun Microsystems, Inc. - Although
computer system 100 contains a single processor, it will be apparent to those skilled in the art that methods consistent with the present invention operate equally as well with a multi-processor environment. -
Memory 130 includes aprogram 110 and aperformance analyzer 115.Program 110 is a multi-threaded program. For purposes of facilitating performance analysis ofprogram 110 in a manner consistent with the principles of the present invention, the program is instrumented with appropriate instructions of the developer's choosing to generate certain performance data. -
Performance analyzer 115 is comprised of two components. Thefirst component 115 a is a library of functions to be performed in a manner specified by the instrumented program. Thesecond component 115 b is a developer interface that is used for two functions: (1) automatically instrumenting a program; and (2) viewing performance information collected when an instrumented program is executed. - As explained, instrumentation can be done automatically with the use of
performance analyzer interface 115 b. According to this approach, the developer simply specifies for the analyzer the type of performance data to be collected and the analyzer adds the appropriate commands from the performance analysis database access language to the program in the appropriate places. Techniques for automatic instrumentation in this manner are familiar to those skilled in the art. Alternatively, the developer may manually insert commands from the performance analysis database access language in the appropriate places in the program so that during execution specific performance data is recorded. The performance data generated during execution ofprogram 110 is recorded in memory, for example,main memory 130. -
Performance analyzer interface 115 b permits developers to view performance information corresponding to the performance data recorded whenprogram 110 is executed. As explained below, the developer may interact with the analyzer to alter the view to display performance information in various configurations to observe different aspects of the program's performance without having to repeatedly execute the program to collect information for each view, provided the program was properly instrumented at the outset. Each view may show (i) a complete measurement cycle for one or more threads; (ii) when each thread enters and leaves each state; and (iii) selected performance criteria corresponding to each state. - Although not shown in FIG. 1, like all computer systems,
system 105 has an operating system that controls its operations, including the execution ofprogram 110 byprocessor 150. Also, although aspects of one implementation consistent with the principles of the present invention are described herein with performance analyzer stored in main memory 120, one skilled in the art will appreciate that all or part of systems and methods consistent with the present invention may be stored on or read from other computer-readable media, such as secondary storage devices, like hard disks, floppy disks, and CD-ROM; a carrier wave received from the Internet; or other forms of ROM or RAM. Finally, although specific components ofdata processing system 100 have been described, one skilled in the art will appreciate that a data processing system suitable for use with the exemplary embodiment may contain additional or different components. - FIG. 2 depicts a block diagram of a performance analysis system consistent with the present invention. As shown,
program 210 consists ofmultiple threads Processor 220 executesthreads Memory 240 represents a shared memory that may be accessed by all executing threads. A protocol for coordinating access to a shared memory is described in U.S. patent application Ser. No. ______, of Shaun Dennie, entitled “Protocol for Coordinating the Distribution of Shared Memory”, (Attorney Docket No. 06502-0207-00000), which is incorporated herein by reference. Although asingle processor 220 is shown, multiple processors may be used to executethreads - To facilitate parallel execution of
multiple threads system partitions memory 240 into segments designated for operations associated with each thread and initializes the field of each segment. For example,memory segment 245 is comprised of enter and exit state identifiers, developer specified information, and thread identification information. An enter state identifier stores data corresponding to when, during execution, a thread enters a particular state. Similarly, an exit state identifier stores data corresponding to when, during execution of an application program, a thread leaves a particular state. Developer specified data represents the performance analysis data collected. - A reserved area of
memory 250 is used to perform administrative memory management functions, such as, coordinating the distribution of shared memory to competing threads. The reserved area ofmemory 250 is also used for assigning identification information to threads using memory. - The flow chart of FIG. 3 provides additional details regarding the operation of a performance analysis system consistent with an implementation of the present invention. Instructions that generate performance data are inserted into a program (step305). The instrumented program is executed and the performance data are generated (
steps 310 and 315). In response to a request to view performance data, performance analyzer accesses and displays the performance data (step 320). - Performance analyzer is capable of displaying both the performance data and the related source code and assembly code, i.e., machine instructions, corresponding to the data. This allows a developer to relate performance data to both the source code and the assembly code that produced the data.
- FIG. 4 shows a
display 400 with two parts labeled A and B, respectively. The first part, labeled A, shows the performance characteristics of an application program in four dimensions: threads, time, states, and performance. Performance information for each thread is displayed horizontally using a bar graph-type format. Time is represented on the horizontal axis; performance is represented on the vertical axis. - Two threads,
thread 1 andthread 2 indisplay 400, were executing concurrently. As shown, the threads began executing at different times. The horizontal axis forthread 1 is labeled 402.Thread 1 began executing at a point in time labeled “x” on thehorizontal axis 402. The horizontal axis forthread 2 is labeled 404.Thread 2 began executing at time “b”. Each thread performed operations in multiple states, each state being represented by a different pattern.Thread 2 was idle at the beginning of the measuring period. One reason for this idle period may be thatthread 2 was waiting for resources fromthread 1. Based on this information, a developer can allocate operations of a thread among states such that performance will be improved, for example, by not executing concurrent operations that require use of the same system resources. - As shown,
thread 1 enteredstate 410 at a point in time “x” on thehorizontal axis 402 and leftstate 410 at time “y”, and enteredstate 420 at time “m” and leftstate 420 at time “n”. The horizontal distance between points “x” and “y” is shorter than the horizontal distance between points “m” and “n”. Therefore,thread 1 operated instate 420 longer than it operated instate 410. The vertical height of the bars show a level of performance. The vertical height forstate 410 is lower than the vertical height forstate 420, showing thatstates - The bottom-half of the display, labeled B, illustrates an expression evaluation feature of the performance analyzer's interface. A developer specifies computational to expressions related to a performance metric of a selected state(s). The performance analyzer computes the value of an expression for the performance data collected.
- In the example shown, the developer has selected
state 440. The expression on the first line, “NUM_OPS/(100000*TIME)”, is an expression for computing the number (in millions) of floating point instructions per second (MFLOPS). The expression on the second line, “2*_CPU_MHZ” calculates a theoretical peak level of performance for a specified state. Performance analyzer may evaluate these two expressions in conjunction to provide quantitative information about a particular state. For example, by dividing MFLOPS by the theoretical peak performance level forstate 440, performance analyzer calculates for the developer the percentage of theoretical peak represented by each operation instate 440. - Conclusion
- Methods and systems consistent with the present invention collect performance data from hardware and software components of an application program, allowing a developer to understand how performance data relates to each thread of a program and complementing a developer's ability to understand and subsequently diagnose performance issues occurring in a program.
- Although the foregoing description has been described with reference to a specific implementation, those skilled in the art will know of various changes in form and detail which may be made without departing from the spirit and scope of the present invention as defined in the appended claims and the full scope of their equivalents.
Claims (9)
Priority Applications (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/244,895 US6434714B1 (en) | 1999-02-04 | 1999-02-04 | Methods, systems, and articles of manufacture for analyzing performance of application programs |
JP2000024065A JP2000235511A (en) | 1999-02-04 | 2000-02-01 | Performance analysis method for application program and system/device used for performance analysis for application program |
EP00400307A EP1026592A3 (en) | 1999-02-04 | 2000-02-03 | Method for analyzing the performance of application programs |
US10/002,885 US20020059544A1 (en) | 1999-02-04 | 2001-11-02 | Methods and systems for determining and displaying activities of concurrent processes |
JP2011264395A JP5432973B2 (en) | 1999-02-04 | 2011-12-02 | Application program performance analysis method, system and apparatus used for application program performance analysis |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/244,895 US6434714B1 (en) | 1999-02-04 | 1999-02-04 | Methods, systems, and articles of manufacture for analyzing performance of application programs |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/002,885 Continuation-In-Part US20020059544A1 (en) | 1999-02-04 | 2001-11-02 | Methods and systems for determining and displaying activities of concurrent processes |
Publications (2)
Publication Number | Publication Date |
---|---|
US20020073360A1 true US20020073360A1 (en) | 2002-06-13 |
US6434714B1 US6434714B1 (en) | 2002-08-13 |
Family
ID=22924533
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/244,895 Expired - Lifetime US6434714B1 (en) | 1999-02-04 | 1999-02-04 | Methods, systems, and articles of manufacture for analyzing performance of application programs |
Country Status (3)
Country | Link |
---|---|
US (1) | US6434714B1 (en) |
EP (1) | EP1026592A3 (en) |
JP (2) | JP2000235511A (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020059503A1 (en) * | 1999-02-04 | 2002-05-16 | Sun Microsystems, Inc. | Protocol for coordinating the distribution of shared memory |
US6802057B1 (en) | 2000-05-03 | 2004-10-05 | Sun Microsystems, Inc. | Automatic generation of fortran 90 interfaces to fortran 77 code |
US6957208B1 (en) * | 2000-10-31 | 2005-10-18 | Sun Microsystems, Inc. | Method, apparatus, and article of manufacture for performance analysis using semantic knowledge |
US20080313571A1 (en) * | 2000-03-21 | 2008-12-18 | At&T Knowledge Ventures, L.P. | Method and system for automating the creation of customer-centric interfaces |
US7533371B1 (en) * | 2003-09-22 | 2009-05-12 | Microsoft Corporation | User interface for facilitating performance analysis for processing |
US7797585B1 (en) * | 2005-05-09 | 2010-09-14 | Emc Corporation | System and method for handling trace data for analysis |
WO2017023324A1 (en) * | 2015-08-06 | 2017-02-09 | Hewlett Packard Enterprise Development Lp | Application component warnings based upon performance ratings |
Families Citing this family (40)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7076784B1 (en) * | 1997-10-28 | 2006-07-11 | Microsoft Corporation | Software component execution management using context objects for tracking externally-defined intrinsic properties of executing software components within an execution environment |
US6593940B1 (en) * | 1998-12-23 | 2003-07-15 | Intel Corporation | Method for finding errors in multithreaded applications |
US7086007B1 (en) | 1999-05-27 | 2006-08-01 | Sbc Technology Resources, Inc. | Method for integrating user models to interface design |
US7224790B1 (en) | 1999-05-27 | 2007-05-29 | Sbc Technology Resources, Inc. | Method to identify and categorize customer's goals and behaviors within a customer service center environment |
US6748555B1 (en) * | 1999-09-09 | 2004-06-08 | Microsoft Corporation | Object-based software management |
US6778643B1 (en) * | 2000-03-21 | 2004-08-17 | Sbc Technology Resources, Inc. | Interface and method of designing an interface |
KR100361340B1 (en) * | 2000-05-15 | 2002-12-05 | 엘지전자 주식회사 | Controlling method for cpu clock |
US6986130B1 (en) | 2000-07-28 | 2006-01-10 | Sun Microsystems, Inc. | Methods and apparatus for compiling computer programs using partial function inlining |
US6681345B1 (en) * | 2000-08-15 | 2004-01-20 | International Business Machines Corporation | Field protection against thread loss in a multithreaded computer processor |
KR100613201B1 (en) * | 2000-08-28 | 2006-08-18 | 마이크로코넥트 엘엘씨 | Measuring method for cpu usage |
JP2002328818A (en) * | 2001-02-27 | 2002-11-15 | Sony Computer Entertainment Inc | Information processor, integrated information processor, method for calculating execution load and computer program |
JP3491617B2 (en) * | 2001-03-01 | 2004-01-26 | 日本電気株式会社 | Operation report creation method, operation report creation method, and operation report creation program |
US7478368B2 (en) * | 2001-04-24 | 2009-01-13 | International Business Machines Corporation | Organization and visualization of performance data in selected display modes |
US7065201B2 (en) * | 2001-07-31 | 2006-06-20 | Sbc Technology Resources, Inc. | Telephone call processing in an interactive voice response call management system |
US7096388B2 (en) * | 2001-08-08 | 2006-08-22 | Avaya Technology Corp. | Fault tolerance software system with periodic external self-test failure detection |
CA2358563A1 (en) * | 2001-10-05 | 2003-04-05 | Ibm Canada Limited - Ibm Canada Limitee | Method and system for managing software testing |
US7305070B2 (en) * | 2002-01-30 | 2007-12-04 | At&T Labs, Inc. | Sequential presentation of long instructions in an interactive voice response system |
US6914975B2 (en) | 2002-02-21 | 2005-07-05 | Sbc Properties, L.P. | Interactive dialog-based training method |
US7272820B2 (en) * | 2002-12-12 | 2007-09-18 | Extrapoles Pty Limited | Graphical development of fully executable transactional workflow applications with adaptive high-performance capacity |
US7664798B2 (en) * | 2003-09-04 | 2010-02-16 | Oracle International Corporation | Database performance baselines |
US7673291B2 (en) * | 2003-09-05 | 2010-03-02 | Oracle International Corporation | Automatic database diagnostic monitor architecture |
US7421621B1 (en) * | 2003-09-19 | 2008-09-02 | Matador Technologies Corp. | Application integration testing |
US7027586B2 (en) | 2003-12-18 | 2006-04-11 | Sbc Knowledge Ventures, L.P. | Intelligently routing customer communications |
JP4237661B2 (en) * | 2004-03-25 | 2009-03-11 | 株式会社東芝 | System LSI design support system and design support method |
US7480834B2 (en) * | 2005-01-06 | 2009-01-20 | International Business Machines Corporation | System and method for monitoring application availability |
US7577875B2 (en) * | 2005-09-14 | 2009-08-18 | Microsoft Corporation | Statistical analysis of sampled profile data in the identification of significant software test performance regressions |
US9323578B2 (en) | 2006-01-19 | 2016-04-26 | International Business Machines Corporation | Analyzing wait states in a data processing system |
US7474991B2 (en) * | 2006-01-19 | 2009-01-06 | International Business Machines Corporation | Method and apparatus for analyzing idle states in a data processing system |
US20080313628A1 (en) * | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Metrics pack distribution for data reporting tool |
US8990811B2 (en) * | 2007-10-19 | 2015-03-24 | Oracle International Corporation | Future-based performance baselines |
US8719801B2 (en) * | 2008-06-25 | 2014-05-06 | Microsoft Corporation | Timing analysis of concurrent programs |
US9691171B2 (en) | 2012-08-03 | 2017-06-27 | Dreamworks Animation Llc | Visualization tool for parallel dependency graph evaluation |
CN102945198B (en) * | 2012-10-19 | 2016-03-02 | 浪潮电子信息产业股份有限公司 | A kind of method characterizing high-performance calculation application characteristic |
US9317349B2 (en) | 2013-09-11 | 2016-04-19 | Dell Products, Lp | SAN vulnerability assessment tool |
US10223230B2 (en) | 2013-09-11 | 2019-03-05 | Dell Products, Lp | Method and system for predicting storage device failures |
US9454423B2 (en) * | 2013-09-11 | 2016-09-27 | Dell Products, Lp | SAN performance analysis tool |
US9720758B2 (en) | 2013-09-11 | 2017-08-01 | Dell Products, Lp | Diagnostic analysis tool for disk storage engineering and technical support |
IN2014MU00158A (en) | 2014-01-16 | 2015-08-28 | Tata Consultancy Services Ltd | |
US9436411B2 (en) | 2014-03-28 | 2016-09-06 | Dell Products, Lp | SAN IP validation tool |
US20170147422A1 (en) * | 2015-11-23 | 2017-05-25 | Alcatel-Lucent Canada, Inc. | External software fault detection system for distributed multi-cpu architecture |
Family Cites Families (49)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0138535A3 (en) | 1983-10-13 | 1987-01-28 | British Telecommunications Plc | Visual display logic simulation system |
US4812996A (en) | 1986-11-26 | 1989-03-14 | Tektronix, Inc. | Signal viewing instrumentation control system |
WO1990014629A2 (en) | 1989-05-26 | 1990-11-29 | Massachusetts Institute Of Technology | Parallel multithreaded data processing system |
US5079707A (en) | 1990-01-19 | 1992-01-07 | The Boeing Company | Integrated certification-calibration system for a testing system having multiple test instruments |
US5325499A (en) | 1990-09-28 | 1994-06-28 | Tandon Corporation | Computer system including a write protection circuit for preventing illegal write operations and a write poster with improved memory |
JPH0774984B2 (en) * | 1991-06-10 | 1995-08-09 | インターナショナル・ビジネス・マシーンズ・コーポレイション | System resource utilization measurement method and data processing system |
JPH0816877B2 (en) * | 1991-06-10 | 1996-02-21 | インターナショナル・ビジネス・マシーンズ・コーポレイション | Method and system for real-time capture and reduction of resource data for data processing system |
GB9123271D0 (en) | 1991-11-02 | 1991-12-18 | Int Computers Ltd | Data processing system |
US5742793A (en) | 1991-12-18 | 1998-04-21 | Intel Corporation | Method and apparatus for dynamic memory management by association of free memory blocks using a binary tree organized in an address and size dependent manner |
US5553235A (en) * | 1992-10-23 | 1996-09-03 | International Business Machines Corporation | System and method for maintaining performance data in a data processing system |
US5675790A (en) | 1993-04-23 | 1997-10-07 | Walls; Keith G. | Method for improving the performance of dynamic memory allocation by removing small memory fragments from the memory pool |
US5519866A (en) | 1993-06-28 | 1996-05-21 | Taligent, Inc. | Method and apparatus of incrementally linking components of a modeled computer program |
US5325533A (en) | 1993-06-28 | 1994-06-28 | Taligent, Inc. | Engineering system for modeling computer programs |
US5748961A (en) | 1993-07-12 | 1998-05-05 | Digital Equipment Corporation | Efficient method and apparatus for compiling and linking modules of computer code in a large software system |
US5500881A (en) | 1993-07-12 | 1996-03-19 | Digital Equipment Corporation | Language scoping for modular, flexible, concise, configuration descriptions |
US5485574A (en) * | 1993-11-04 | 1996-01-16 | Microsoft Corporation | Operating system based performance monitoring of programs |
US5845310A (en) * | 1993-12-15 | 1998-12-01 | Hewlett-Packard Co. | System and methods for performing cache latency diagnostics in scalable parallel processing architectures including calculating CPU idle time and counting number of cache misses |
US5636374A (en) | 1994-01-04 | 1997-06-03 | Intel Corporation | Method and apparatus for performing operations based upon the addresses of microinstructions |
US5963975A (en) | 1994-04-19 | 1999-10-05 | Lsi Logic Corporation | Single chip integrated circuit distributed shared memory (DSM) and communications nodes |
CA2147036A1 (en) * | 1994-05-16 | 1995-11-17 | Yih-Farn Robin Chen | System and method for selective regression testing |
US5724262A (en) | 1994-05-31 | 1998-03-03 | Paradyne Corporation | Method for measuring the usability of a system and for task analysis and re-engineering |
US5613063A (en) | 1994-07-01 | 1997-03-18 | Digital Equipment Corporation | Method and apparatus for checking validity of memory operations |
US5689712A (en) | 1994-07-27 | 1997-11-18 | International Business Machines Corporation | Profile-based optimizing postprocessors for data references |
EP0703534B1 (en) | 1994-09-19 | 2002-05-02 | Siemens Aktiengesellschaft | Computer memory management system |
US5675802A (en) | 1995-03-31 | 1997-10-07 | Pure Atria Corporation | Version control system for geographically distributed software development |
US5581696A (en) | 1995-05-09 | 1996-12-03 | Parasoft Corporation | Method using a computer for automatically instrumenting a computer program for dynamic debugging |
US6101525A (en) | 1995-05-19 | 2000-08-08 | Emc Corporation | Method and apparatus for shared memory cleanup |
US6016474A (en) | 1995-09-11 | 2000-01-18 | Compaq Computer Corporation | Tool and method for diagnosing and correcting errors in a computer program |
US5784698A (en) | 1995-12-05 | 1998-07-21 | International Business Machines Corporation | Dynamic memory allocation that enalbes efficient use of buffer pool memory segments |
US5850554A (en) | 1995-12-29 | 1998-12-15 | Intel Corporation | Compiler tool set for efficiently generating and easily managing multiple program versions of different types |
US5805795A (en) * | 1996-01-05 | 1998-09-08 | Sun Microsystems, Inc. | Method and computer program product for generating a computer program product test that includes an optimized set of computer program product test cases, and method for selecting same |
US5867649A (en) | 1996-01-23 | 1999-02-02 | Multitude Corporation | Dance/multitude concurrent computation |
US5978892A (en) | 1996-05-03 | 1999-11-02 | Digital Equipment Corporation | Virtual memory allocation in a virtual address space having an inaccessible gap |
US6058460A (en) | 1996-06-28 | 2000-05-02 | Sun Microsystems, Inc. | Memory allocation in a multithreaded environment |
US5787480A (en) | 1996-07-17 | 1998-07-28 | Digital Equipment Corporation | Lock-up free data sharing |
US5905488A (en) | 1996-10-11 | 1999-05-18 | Xerox Corporation | Local inking with gray pixels |
US5835705A (en) * | 1997-03-11 | 1998-11-10 | International Business Machines Corporation | Method and system for performance per-thread monitoring in a multithreaded processor |
US6044438A (en) | 1997-07-10 | 2000-03-28 | International Business Machiness Corporation | Memory controller for controlling memory accesses across networks in distributed shared memory processing systems |
US5872977A (en) | 1997-08-08 | 1999-02-16 | International Business Machines Corporation | Object-oriented method and apparatus for creating a makefile |
US5974536A (en) | 1997-08-14 | 1999-10-26 | Silicon Graphics, Inc. | Method, system and computer program product for profiling thread virtual memory accesses |
GB9717718D0 (en) | 1997-08-22 | 1997-10-29 | Philips Electronics Nv | Memory management with compaction of data blocks |
US5991893A (en) | 1997-08-29 | 1999-11-23 | Hewlett-Packard Company | Virtually reliable shared memory |
US6065019A (en) | 1997-10-20 | 2000-05-16 | International Business Machines Corporation | Method and apparatus for allocating and freeing storage utilizing multiple tiers of storage organization |
US6018793A (en) | 1997-10-24 | 2000-01-25 | Cirrus Logic, Inc. | Single chip controller-memory device including feature-selectable bank I/O and architecture and methods suitable for implementing the same |
US6098169A (en) * | 1997-12-23 | 2000-08-01 | Intel Corporation | Thread performance analysis by monitoring processor performance event registers at thread switch |
US5963726A (en) | 1998-03-20 | 1999-10-05 | National Instruments Corporation | Instrumentation system and method including an improved driver software architecture |
US6249906B1 (en) * | 1998-06-26 | 2001-06-19 | International Business Machines Corp. | Adaptive method and system to minimize the effect of long table walks |
US6978357B1 (en) | 1998-07-24 | 2005-12-20 | Intel Corporation | Method and apparatus for performing cache segment flush and cache segment invalidation operations |
US6269457B1 (en) * | 2000-06-01 | 2001-07-31 | Testing Technologies, Inc. | Technology regression and verification acceptance method |
-
1999
- 1999-02-04 US US09/244,895 patent/US6434714B1/en not_active Expired - Lifetime
-
2000
- 2000-02-01 JP JP2000024065A patent/JP2000235511A/en active Pending
- 2000-02-03 EP EP00400307A patent/EP1026592A3/en not_active Withdrawn
-
2011
- 2011-12-02 JP JP2011264395A patent/JP5432973B2/en not_active Expired - Lifetime
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020059503A1 (en) * | 1999-02-04 | 2002-05-16 | Sun Microsystems, Inc. | Protocol for coordinating the distribution of shared memory |
US20080313571A1 (en) * | 2000-03-21 | 2008-12-18 | At&T Knowledge Ventures, L.P. | Method and system for automating the creation of customer-centric interfaces |
US8131524B2 (en) | 2000-03-21 | 2012-03-06 | At&T Intellectual Property I, L.P. | Method and system for automating the creation of customer-centric interfaces |
US6802057B1 (en) | 2000-05-03 | 2004-10-05 | Sun Microsystems, Inc. | Automatic generation of fortran 90 interfaces to fortran 77 code |
US6957208B1 (en) * | 2000-10-31 | 2005-10-18 | Sun Microsystems, Inc. | Method, apparatus, and article of manufacture for performance analysis using semantic knowledge |
US7533371B1 (en) * | 2003-09-22 | 2009-05-12 | Microsoft Corporation | User interface for facilitating performance analysis for processing |
US7797585B1 (en) * | 2005-05-09 | 2010-09-14 | Emc Corporation | System and method for handling trace data for analysis |
WO2017023324A1 (en) * | 2015-08-06 | 2017-02-09 | Hewlett Packard Enterprise Development Lp | Application component warnings based upon performance ratings |
Also Published As
Publication number | Publication date |
---|---|
JP5432973B2 (en) | 2014-03-05 |
JP2000235511A (en) | 2000-08-29 |
EP1026592A3 (en) | 2004-05-12 |
US6434714B1 (en) | 2002-08-13 |
EP1026592A2 (en) | 2000-08-09 |
JP2012094166A (en) | 2012-05-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6434714B1 (en) | Methods, systems, and articles of manufacture for analyzing performance of application programs | |
Miller et al. | IPS-2: The second generation of a parallel program measurement system | |
US8627335B2 (en) | Method and apparatus for data space profiling of applications across a network | |
US7178134B2 (en) | Method and apparatus for resolving memory allocation trace data in a computer system | |
US8640114B2 (en) | Method and apparatus for specification and application of a user-specified filter in a data space profiler | |
US8813055B2 (en) | Method and apparatus for associating user-specified data with events in a data space profiler | |
Goldberg et al. | Mtool: An integrated system for performance debugging shared memory multiprocessor applications | |
Lim et al. | Waiting algorithms for synchronization in large-scale multiprocessors | |
US20080127149A1 (en) | Method and Apparatus for Computing User-Specified Cost Metrics in a Data Space Profiler | |
US20080127120A1 (en) | Method and apparatus for identifying instructions associated with execution events in a data space profiler | |
JPH0833845B2 (en) | Software motion analyzer | |
Feljan et al. | Towards a model-based approach for allocating tasks to multicore processors | |
CN109542731B (en) | A kind of performance monitoring method of the progressive lower brill of level towards GPU | |
US20020059544A1 (en) | Methods and systems for determining and displaying activities of concurrent processes | |
Karavanic et al. | Improving online performance diagnosis by the use of historical performance data | |
US11768754B2 (en) | Parallel program scalability bottleneck detection method and computing device | |
JPH09212385A (en) | Parallel program debugging device | |
JPH0683608A (en) | Program analysis support device | |
de Kergommeaux et al. | Monitoring parallel programs for performance tuning in cluster environments | |
Lee et al. | Experiences with a parallel algorithm for data flow analysis | |
Rajamony et al. | Performance debugging shared memory parallel programs using run-time dependence analysis | |
JPH11242614A (en) | Device and method for real-time verification for parallel programs | |
Grabner et al. | Debugging of concurrent processes | |
Broberg et al. | Performance optimization using extended critical path analysis in multithreaded programs on multiprocessors | |
Smyk et al. | Global application states monitoring applied to graph partitioning optimization |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SUN MICROSYSTEMS, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LEWIS, BRADLEY;WEEK, JEREMY;BOUCHER, MICHAEL;AND OTHERS;REEL/FRAME:009920/0017 Effective date: 19990421 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FEPP | Fee payment procedure |
Free format text: PAYER NUMBER DE-ASSIGNED (ORIGINAL EVENT CODE: RMPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
FPAY | Fee payment |
Year of fee payment: 8 |
|
FPAY | Fee payment |
Year of fee payment: 12 |
|
AS | Assignment |
Owner name: ORACLE AMERICA, INC., CALIFORNIA Free format text: MERGER AND CHANGE OF NAME;ASSIGNORS:ORACLE USA, INC.;SUN MICROSYSTEMS, INC.;ORACLE AMERICA, INC.;REEL/FRAME:037278/0612 Effective date: 20100212 |