Welcome to March’s Progress Report! If you were left wanting more after last month’s report, then your wait is over. March saw massive strides in all facets of the emulator as multiple work-in-progress pull requests from our regular developers were finally completed and merged. To start things off, Nekotekina implemented a new LLVM-based SPU interpreter, updated the LLVM submodule to version 9 and fixed the “Out of memory” errors faced by some games. On the other hand, kd-11 continued his mission to improve the RSX texture handling which fixed issues faced in over two dozen titles while GalCiv implemented native support for the DualShock 3. Finally, elad335 made a host of improvements to various core components improving performance and accuracy in an assortment of titles, most notably God of War 3, when running on Windows.
In addition to the following report, further details of Nekotekina and kd-11’s work during March and upcoming contributions can be found in their weekly reports on Patreon. This month’s Patreon reports are:
Status update from kd-11 (2019-03-02)
Status update from Nekotekina (2019-03-05)
Status update from kd-11 (2019-03-18)
Status update from Nekotekina (2019-03-30)
Table of Contents
Major Improvements
Games
Other Improvements
Conclusion
This month saw more maintenance work done to the compatibility list where duplicate game IDs for the same game were merged into one single entry. A grand total of 118 threads were merged allowing for a fairer representation of RPCS3’s compatibility at this time. Adding to this, our community of testers continued their efforts in acquiring and testing obscure titles that had not been tested recently. The results of this revealed that many of the titles in the Ingame and Intro categories were now indeed Playable and moved according. Thanks to the dual effect of both efforts, the Playable category saw a marginal increase while the Ingame and Intro categories saw a sharp decrease in total number of games listed. Finally, thanks to elad335’s improvements to sys_vm, Doom 3: BFG Edition moved from Nothing category straight into the Playable category, leaving just two more titles in the Nothing category! For a more detailed look, you can view the compatibility history page to see exactly which games had their status changed this month.
On Git statistics, there have been 6331 lines of code added and 2573 removed through 105 commits by 9 authors.
Major RPCS3 Improvements
RSX Texturing improvements (#5675, #5726, #5741, #5752)
This month, kd-11 was focused on work that aimed to properly preserve rasterized data without falling back to slow synchronization techniques. From the roadmap shared with patreons, the first step towards that is to build the foundation for smooth implementation of features by rewriting the texture cache to improve how texture data is interpreted for sampling. This was an arduous task as past experience showed us that many games are extremely sensitive to texture cache changes and often regress when modified. However, with the help of our dedicated tester base testing every game at their disposal, kd-11 was able to rewrite the texture cache and seamlessly address most regressions. For those looking for a detailed look into what the texture cache is and how it works in RPCS3, you can read our “Quick” Primer on Texture Caching in RPCS3.
Over six weeks, kd-11 hammered the texture cache into shape by going for an accuracy focused approach with regards to advanced synchronization and data resolve across the separate surface management cache and the texture data cache. The primary objective was to improve the texture search logic by attempting to ‘merge’ the surface store data and the texture cache data whenever overlaps are encountered. Due to performance concerns, a full data collapse would be infeasible but since the two sources are optionally data locked, this should almost always resolve correctly especially when Write Color Buffers is enabled.
 |
 |
After this, he immediately began working on getting the blit engine to behave correctly with the new structures and policies implemented over the rest of the cache. The blit engine is the 2D rendering engine inside the GPU, capable of handling memory-intensive tasks such as asynchronous memory copies over DMA, texture scaling, format conversion, fast texture data copies as well as rotations around the X or the Y axis which can be thought of as ‘flipping’ the image. This work builds upon the previously existing work as the initial implementation broke many games when using the hardware (GPU) to accelerate these operations, necessitating the use of the ‘CPU blit’ debug option to get the correct output. Apart from this, few more features such as pitch compatibility methods were also implemented but have not been enabled yet. These features will be enabled down the road once the rest of the code is stable enough to accommodate it.
All of this paid off with graphical improvements across a significant number of games and fixed long-standing issues such as flickering in Demon’s Souls, water reflections in Ratchet & Clank: A Crack in Time as well as the Atlantica stage and Genie Jafar stage in Kingdom Hearts HD 1.5 ReMIX (removing the need for Strict Rendering Mode and allowing for resolution upscaling). Games such as Beyond: Two Souls and inFamous: Festival of Blood no longer need CPU blit to properly render textures leading to a significant performance boost as well. Finally, with these changes, Saints Row: The Third and Saints Row IV render graphics for the first time in RPCS3.
 |
 |
||
 |
 |
||
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
New LLVM-based SPU Interpreter & SPU improvements (#5762, #5721, #5770)
This month, Nekotekina overhauled the SPU Interpreter (fast) into an LLVM-based Interpreter that shares the same backend as the current LLVM Recompiler. This was an important step as the unified the backend for the interpreter and recompiler eliminates any accuracy differences between the two implementations and also simplifies the underlying code. Added to this, the interpreter now respects the xfloat accuracy options that were previously restricted only to the LLVM Recompiler and can also benefit from architecture-specific code generation by LLVM.
In terms of performance, the new interpreter saw massive performance improvements over the legacy implementation with some games titles achieving a 20% increase in FPS. Furthermore, in some titles such Sengoku Musou 4 and Dynasty Warriors: Gundam 3, the FPS when using the interpreter improved performance from 10FPS to 50FPS, surpassing even the SPU ASMJIT Recompiler!
This was the first step in Nekotekina’s SPU/PPU roadmap given to patreons and he is hard at work improving the synergies between the two backends and using it to yield much greater performance and accuracy. If you wish read the full PPU/SPU roadmap by Nekotekina, you can check his patreon post.
Besides this, Nekotekina also improved the MMIO method handling in the emulator. During debugging, elad335 identified that SPU LLVM emits some MFC code in CPUs with TSX enabled and some checks for address in SPU MFC are MMIO but treated as usual DMA leading to issues. In the context of SPU MFC, DMA (Direct Memory Access) is a method used to read/write data from/to the SPU local cache into the global memory. Whereas, MMIO (Memory-mapped I/O) is used to read/write to a register and also perform an action in addition to data access. To prevent SPU LLVM from incorrectly using DMA for these checks, Nekotekina implemented a fallback to the general MFC MMIO handler that will correctly handle such transactions. Not only does this improve the accuracy of emulation, but in cases like raw SPU MMIO, the check was previously handled by the PPU access violation handler which is known to be expensive. Now, these checks also use the general MFC MMIO handler leading to a notable speed increase.
Finally, Nekotekina also bumped the version of LLVM used in RPCS3 from 7.0 to 9.0 and implemented use of few new features offered by the latest version. While LLVM 9.0 is still in the research phase and has not been fully released, Nekotekina opted to adopt it early and utilise the benefits that it provides.
Support for DualShock 3 controllers (#5641)
As with every PlayStation console, the PlayStation 3 was released along with a new DualShock controller, the DualShock 3. This controller was a great leap forward as not only did it still have pressure-sensitive buttons from its predecessor, it also introduced motion control technology (SIXAXIS as Sony called it). This was a large step ahead from other controllers that lacked pressure-sensitive buttons and SIXAXIS but the exclusivity of these features also led to rather low adoption-rate from game developers. That’s not to say they weren’t used, many first party or PlayStation 3 exclusives made use of these features such as Metal Gear Solid 4, Flower, Heavy Rain, Uncharted 1 and Heavenly Sword.
Since the DualShock 3 was made for the PlayStation 3, its drivers are incompatible on PC and need third-party tools to work. While most third-party tools used to extend support for the DualShock 3 worked well with RPCS3, their implementations caused some notable issues. Most commonly used tools behave as wrappers around the standard XInput API on Windows, converting input from the DualShock 3 to that which is compatible with XInput. However, since XInput had no functionality for motion controls or pressure-sensitive buttons, such inputs from the controller are simply ignored. This called for a dedicated pad handler for DualShock 3 to effectively support its full range of features.
Now, anyone who has followed RPCS3 knows that it has a dedicated pad handler for the DualShock 4 since May 2017. The DualShock 4 shared a lot of similarities with its predecessor such as being designed to work solely with the PlayStation 4, having a range of unique features not present in other controllers and requiring third-party software to work on PC. So why was the implementation of the native pad handler much simpler compared to the DualShock 3? To put it simply, the DualShock 3 suffers from certain fundamental differences that prevent it from being supported as easily.
Most controllers interact with host systems through the USB port and adhere to the USB HID (Human Interface Device) class defined by the USB Implementers Forum. As per this specification, every device describes how it will communicate with the host system to allow the host system to accurately predict and define all current and future interactions with the device. During enumeration the device describes how its reports are to be structured so that the host system can properly prepare to receive this information. The creation of this standard enables Operating Systems to implement a standardised framework for HIDs and allows for seamless integration with various devices. Similarly, the Bluetooth HID class is a simple wrapper around the USB HID class simplifying OS implementation. So if you ever wondered who thank when your printer, mouse and keyboard just work when connected for the first time, now you know!
All controllers (including the DualShock 4) adhere to this USB HID class and communicate with consoles/PCs using this specification. However, the DualShock 3 deviates from the specification and requires the host system to send a GET_REPORT request to start communication. Since the report is not advertised by the DualShock 3, the OS does not allow us to send this request to it. To tackle this deficiency, GalCiv implemented a system where the controller is first recognised by the OS as a generic USB device and then subsequently handled by RPCS3 as a controller to allow input to games. While this approach has allowed us to support DualShock 3 and all of its unique features, it does come with a few requirements. Firstly, users are required to install generic USB drivers (like WinUSB) to access the controller. Since the controller is required to be recognised as a USB device, Bluetooth connectivity is not supported at this moment. Finally, for the time being, native support for DualShock 3 is restricted only to Windows. With this, RPCS3 finally natively supports the DualShock 3 allowing for better emulation of titles that utilised the controllers unique offerings!
sys_fs (fsync) Improvements (#5506)
Continuing his bug fixing spree, elad335 made a wide variety of improvements to the emulator this month. Only 5 pull requests were made but each of these contained multiple fixes for a wide range of components. While most of these fixes are significant, not all translate directly to games; some improvements can be seen in multiple titles while some fix specific issues to a single game but others don’t show any immediate effects in games. If we were to list all these changes and explain why they were made, this report would never see the light of day. So here’s a quick look at one fascinating change made by elad335 this month. For the full list of improvements, see the Other Improvements section below.
While debugging a few AAA titles, a startling discovery was made. Two users tested God of War 3 with similar hardware but one user barely managed to hit 20FPS while the other was comfortably at 40FPS. Investigating why such a drastic discrepancy existed on similar hardware, a diligent check uncovered that one user had the game on an SSD while the other had it on a HDD. Now this may seem reasonable at first glance, it is anything but. The PlayStation 3’s Blu-ray drive operated at a maximum speed of 9 MB/s while the internal hard drive was able to transfer data at upto 100 MB/s (depending on the model). Both these numbers are much lower than PC HDDs and leagues below SSDs. This is why the users barely find any difference in performance when running game on RPCS3 from either a HDD or an SSD. Even the cache generated by RPCS3 is loaded into the RAM/VRAM while booting games nullifying any effect in this regard. So why did God of War 3 require an SSD to operate at maximum performance?
A big clue to answering this question came when one of the testers decided to run the game on Linux (since all previous metrics were obtained only on Windows). Running God of War 3 on Linux, we were easily able to reach the 40FPS mark at the test spot, only this time the game was still running on a HDD. This result immediately pointed to some special handling in Windows for HDDs as opposed to a bug in RPCS3. God of War 3 was clearly triggering something that was handled easily on Linux but choked Windows with the same HDD. Investigating further, elad335 identified that the issue was due to frequent file cache buffers flushing by the game.
What does file cache buffers flushing do you ask? Generally, an operating system stores data in file read and write operations in system-maintained data buffers to optimise disk performance. When an application writes to a file, the system usually buffers the data and writes the data to the disk on a regular basis. However, an application can force the operating system through an fsync transfer to flush contents of these data buffers to the disk. Some HDDs also have dedicated in-built cache for further buffering and Windows goes an extra step and requires the disk to clear this cache as well. Buffers are flushed mainly to ensure a consistent state of written files in the case of power loss events and similar disasters.
Coming back to the problem at hand, for some reason (best known only to SCE), God of War 3 requests for a file buffer flush after every 1MB written to disk. This was handled fairly well on Linux but proved expensive on Windows and at such a high volumes, crippled performance as PPU threads would wait for this operation to complete before yielding. elad335 addressed this by yielding the PPU thread before performing the buffer flush so the emulator was free to reschedule the thread immediately for the next task. This solution brought parity in performance between both operating systems.
 |
 |
VSH Progress
We’re wrapping up this month’s coverage with a quick update on running VSH in RPCS3. Back in November of last year, we showcased work done by ruipin and many developers to reach XMB menu (PlayStation 3’s GUI) in RPCS3. At the time, users were required to dump xRegistry.sys from their PlayStation 3 to allow VSH to boot. As it turns out, ruipin has been chipping away at VSH to see what makes it tick. After extensive reverse engineering, he finally managed to implement the sys_config syscalls, one of the key features required for RPCS3 to be able to boot VSH standalone. VSH uses sys_config as a way to permanently store system settings and hardware configuration, e.g. how many controllers are connected. Without these syscalls implemented, VSH is unable to identify connected controllers, and as such does not recognize any input.
With this advancement made in the VSH branch, RPCS3 can now boot VSH and proceed past the initial system configuration and to XMB without the xRegistry.sys file. While the VSH branch is not yet ready to be merged into the master branch, the work done on sys_config is mostly complete and ruipin has opened a pull request to have the same merged into master. For those of you interested in the technical description of sys_config and its implementation, click here.
If you’re interested to check out the XMB menu running on RPCS3, refer to the instructions here. Please note that while the developers have made large strides to boot VSH, it still has a long way to go before it is ready for endusers. In its current state, it cannot be used to boot games installed in RPCS3. However, PlayStation 3 themes that are installed in RPCS3, can be applied and used in the XMB menu!
Games
Gran Turismo 6 & HD Concept
This month saw a large number of racing titles improve in performance and stability and with the PlayStation 3, you can’t mention racing titles without the Gran Turismo series. Since first going ingame in November 2018, the Gran Turismo series has been steadily improving in the graphics and stability departments. Thanks to the efforts from various developers in the past few months and elad335’s implementation of SPU page faults this month, Gran Turismo HD Concept has finally become playable! Check out the extended gameplay footage below:
And if that wasn’t enough, thanks to the page faults notification fixes by elad335, Gran Turismo 6 (and the Ayrton Senna Special Edition) now progresses ingame as well! However, being the latest instalment in the series on the PlayStation 3, it does currently suffer from heavy graphical issues. The game also suffers from low performance due to requiring PPU Interpreter to progress ingame. However, work has already been undertaken to improve compatibility of PPU LLVM which will significantly boost performance in such titles.
Supercar & Ferrari series
Multiple racing titles from Eutechnyx studio went in-game for the first time in January. Thanks to the stability improvements made to the emulator in the past two months, Absolute Supercars, Supercar Challenge, Ferrari Challenge: Trofeo Pirelli and Ferrari: The Race Experience are now comfortably playable!
 |
 |
 |
 |
Ridge Racer 7
Continuing the racing title improvements, the PlayStation 3 exclusive Ridge Racer 7 is now Playable! Previously, this title suffered from low performance and a few graphical issues that have now been fixed. Hopefully the first of many Ridge Racer titles to become playable on RPCS3.
NASCAR Unleashed
To finish off the racing coverage in this month’s report, NASCAR Unleashed also made its way to playable. This console exclusive had escaped testing for over a year and previous only displayed a black-screen when ingame. These issues have now been summarily fixed.
 |
 |
Doom 3: BFG Edition
As most of you are already aware, Doom 3: BFG Edition was one of the three notorious titles still stuck in the Nothing category in RPCS3’s compatibility list. Thanks to improvements made to sys_vm by elad335, this title went from Nothing straight to Playable!
 |
 |
 |
College Hoops 2K7
Moving on to sports titles, College Hoops 2K7 went ingame for the first time this month. While the title has good graphics, it suffers from low performance causing issues with the game physics keeping it from being playable on the emulator.
Move Street Cricket I & II
Finally, this month saw both instalments of Move Street Cricket make their way to playable. If you’re a fan of cricket, these games are definitely worth trying on RPCS3.
 |
 |
Other Improvements
There have been numerous other pull requests merged during the month that couldn’t make it to the Major Improvements section. We have collected a list of all such improvements here, and attached a brief overview to each. Make sure to check out the links provided for them if you are interested, as their GitHub pages usually uncover further details as well as the code changes themselves. To see this whole list right on GitHub, click here.
Nekotekina
5704 – Fixed bug relating to sys_fs_opendir on root directory; Updated sys_lwmutex_lock and sys_lwmutex_unlock (liblv2 HLE) to implement missing SYS_SYNC_RETRY logic; Improve LLVM recognition of Intel Cascade Lake CPUs;
5715 – Updated third-party submodules such as pugixml, xxHash, yaml-cpp and libpng along with other minor optimisations;
5721 – Fixed handling of SPU MFC code in certain scenarios when using the SPU LLVM recompiler, see coverage in major improvements here;
5735 – Fixed a regression caused by PR 5599 where certain games would crash with an Out of memory error when using the SPU ASMJIT recompiler;
5762 – Introduced new LLVM-based SPU interpreter (fast), see coverage in major improvements here;
5770 – Updated LLVM submodule from version 7.0 to version 9.0, see coverage in major improvements here;
kd-11
5675 – Improved RSX texture search logic, see coverage in major improvements here;
5726 – Improved implementation of texture cache and blit engine, see coverage in major improvements here;
5741 – Improved implementation of texture cache, see coverage in major improvements here;
5752 – Improved implementation of texture cache, see coverage in major improvements here.
elad335
5677 – Fixed regression affecting RSX capture replay and improved accuracy of sys_rsx error checking;
5697 – Fixed PPU scheduling in cellSaveData when an external GUI function calls to show saves list allowing games to continue background music and graphics while showing the saves overlay. Also fixed a few deadlocks after loading or storing saves in games such as Grand Theft Auto: San Andreas and Transformers; Fixed a race condition in Vulkan ZCull management regarding access violation handler. This addressed instability issues seen in titles from the Yakuza series when ZCull is enabled; Fixed a race condition in sys_event_flag_set regarding setting flags; Fixed unknown/unimplemented X8R8G8B8 format in image_in using A8R8G8B8 handler; Fixed a regression in Gran Turismo 5 from the above pull request by further improving sys_rsx error checking accuracy by fixing page flags on mirrored memory;
5506 – Yield before flushing io buffers in fsync (sys_fs), see coverage in major improvements here;
5668 – Implemented SPU page fault notifications and fixed various aspects related to page faults such Implementing both RawSPU and threaded SPU page fault recovery, guarding page_fault_notification_entries access with a mutex, adding missing lock in sys_ppu_thread_recover_page_fault/ get_page_fault_context, fixing EINVAL check in sys_ppu_thread_recover_page_fault, previously when the event was not found begin() was erased returning CELL_OK and many more. This fixed the save issue in Gran Turismo 5 and allows Gran Turismo 6 to go ingame for the first time on RPCS3!
5719 – Fixed memory statistics changes of sys_vm, allowing memory statistics to increase or decrease according to action, implemented ENOMEM (not enough memory) and other error checks for sys_vm syscalls. This change fixed Doom 3: BFG Edition allowing it to become playable! Fixed a leak in cellVdec handle management that caused games to crash after 256 cutscenes or other video playback using cellVdec; Fixed segfault on viewing stack memory addresses on the debugger; Fixed picture information packet status check of cellVdecGetPicItem. This fixes Toy Home and SIREN: Blood Curse, allowing both titles to now be fully playable! Fixed a corner case in sys_process_exit2 (exitspawn) where SPUs use the access violation handler and are unable to quit from it. Fixed deadlock in the game selector of Yakuza 1 & 2 HD; Fixed timer state after event queue attached to the timer has been destroyed externally; Fixed page faults notification range check; Fixed flags of device set in cellPadGetInfo; Fixed a race condition in sys_timer_destroy that can result in a deadlock where the timer thread times-out at the same time; Optimized RSX FIFO registers control and cellPad functions; Fixed PPU scheduling in sys_timer when no sleep time has been specified.
GalCiv (RipleyTom)
5696 – As mentioned in last month’s progress report, some of the latest Guitar Hero games check the VendorID/ProductID to determine whether the connected peripheral is a Guitar Hero guitar controller or drum kit. This commit specifies the same through cellPadGetInfo;
5718 – Fixed an issue with lf_queue destructor which caused crashes under certain conditions;
5641 – Implemented a native DualShock 3 pad handler in RPCS3. See coverage in major improvements here;
scribam
5756 – Added function names for HLE cellNetAoi and sceNpMatchingInt modules;
5759 – Added function names for HLE cellDtcpIpUtility, cellPesmUtility and cellVideoPlayerUtility modules;
5761 – Added function names for HLE cellSysutilAvcExt module;
5765 – Added function names for sceNpEulaAbort function;
5767 – Added function names for HLE libad_async and libad_core modules;
5771 – Added function names for HLE cellFontInitLibraryFreeType, cellSysmoduleUnloadModuleInternal and cellSysmoduleLoadModuleInternal modules;
AniLeo
5716 – Fixed a fatal error that occured when installing custom firmware without \n in the version string.
RainbowCookie32
5734 – Updated the tooltip in the Pad configuration dialog to reflect native support for DualShock 3 controllers.
jbeich
5738 – Unbreak build on FreeBSD.
Jocker666z
5774 – Added qtdeclarative5-dev to fix compilation on Debian and Ubuntu.
Closing Words
If you like in-depth technical reports, early access to information, or you simply want to contribute, consider becoming a patron! All donations are greatly appreciated. RPCS3 now has two full-time coders that greatly benefit from the continued support of over 800 generous patrons.
We’re always looking for dedicated writers to help us write these reports. If you have the skill, time and are willing to help, please apply here. Also, come check out our YouTube channel and Discord to stay up-to-date with any big news.
This report was written by HerrHulaHoop, MarioSonic2987, elad335, GalCiv and Digitaldude555.