https://wiki.haskell.org/api.php?action=feedcontributions&user=DonStewart&feedformat=atomHaskellWiki - User contributions [en]2024-03-19T02:48:19ZUser contributionsMediaWiki 1.35.5https://wiki.haskell.org/index.php?title=Haskell_in_industry&diff=55450Haskell in industry2013-02-17T17:35:15Z<p>DonStewart: /* Haskell in Industry */</p>
<hr />
<div>__NOTOC__<br />
<br />
Haskell has a diverse range of use commercially, from aerospace and defense, to finance, to web startups, hardware design firms and a lawnmower manufacturer. This page collects resources on the industrial use of Haskell.<br />
<br />
* The main user conference for industrial Haskell use is CUFP - the [http://cufp.org/ Commercial Users of Functional Programming Workshop].<br />
* The [http://industry.haskell.org Industrial Haskell Group] supports commercial users.<br />
* [http://fpcomplete.com/ FP Complete] is dedicated to the widespread adoption of modern Functional Programming technology, with a focus on the Haskell system. <br />
<br />
== Haskell in Industry ==<br />
<br />
Many companies have used Haskell for a range of projects, including:<br />
<br />
* [http://cufp.galois.com/2007/abstracts.html#CyrilSchmidt ABN AMRO] Amsterdam, The Netherlands<br />
<blockquote><br />
ABN AMRO is an international bank headquartered in Amsterdam. For its<br />
investment banking activities it needs to measure the counterparty risk<br />
on portfolios of financial derivatives. </blockquote><br />
::ABN AMRO's [http://cufp.galois.com/2007/abstracts.html#CyrilSchmidt CUFP talk].<br />
<br />
* Aetion Technologies LLC, Columbus, Ohio<br />
<blockquote><br />
Aetion was a defense contractor in operation from 1999 to 2011, whose applications use artificial intelligence. Rapidly changing priorities make it important to minimize the code impact of changes, which suits Haskell well. Aetion developed three main projects in<br />
Haskell, all successful. Haskell's concise code was perhaps most important for<br />
rewriting: it made it practicable to throw away old code occasionally. DSELs<br />
allowed the AI to be specified very declaratively. <br />
</blockquote><br />
::Aetion's [http://cufp.galois.com/2006/slides/GaryMorris.pdf CUFP talk].<br />
<br />
* Alcatel-Lucent<br />
<blockquote><br />
A consortium of groups, including Alcatel-Lucent, have used Haskell to prototype narrowband software radio systems, running in (soft) real-time.<br />
</blockquote><br />
::Alcatel-Lucent's [http://cufp.org/conference/sessions/2011/fourteen-days-haskell-real-time-programming-projec CUFP talk]<br />
<br />
* [http://www.allstontrading.com/ Allston Trading]<br />
<blockquote><br />
Headquartered in Chicago, Illinois, Allston Trading, LLC is a premier high frequency market maker in over 40 financial exchanges, in 20 countries, and in nearly every conceivable product class. Allston makes some use of Haskell for their trading infrastructure.<br />
</blockquote><br />
<br />
* [http://www.alphaheavy.com/ Alpha Heavy Industries]<br />
<blockquote><br />
Alpha Heavy Industries is an alternative asset manager dedicated to producing superior returns through quantitative methods. They use Haskell as their primary implementation language.<br />
</blockquote><br />
<br />
* [http://www.amgen.com/ Amgen] Thousand Oaks, California<br />
<blockquote><br />
Amgen is a human therapeutics company in the biotechnology industry. Amgen pioneered the development of novel products based on advances in recombinant DNA and molecular biology and launched the biotechnology industry’s first blockbuster medicines.<br />
<br />
Amgen uses Haskell;<br />
<br />
* To rapidly build software to implement mathematical models and other complex, mathematically oriented applications<br />
* Provide a more mathematically rigorous validation of software<br />
* To break developers out of their software development rut by giving them a new way to think about software.<br />
</blockquote><br />
::Amgen's [http://cufp.galois.com/2008/abstracts.html#BalabanDavid CUFP talk].<br />
<br />
* [http://www.ansemond.com/ Ansemond LLC]<br />
<blockquote><br />
"Find It! Keep It! is a Mac Web Browser that lets you keep the pages you<br />
visit in a database. A list of these pages is shown in the 'database<br />
view'. "<br />
</blockquote><br />
<br />
* [http://antiope.com/ Antiope] Fair Haven, New Jersey<br />
<blockquote><br />
Antiope Associates provides custom solutions for wireless communication<br />
and networking problems. Our team has expertise in all aspects of<br />
wireless system design, from the physical and protocol layers to complex<br />
networked applications. Antiope Associates's relies on a number of<br />
advanced techniques to ensure that the communication systems we design<br />
are reliable and free from error. We use custom simulation tools<br />
developed in Haskell, to model our hardware designs..<br />
</blockquote><br />
::Antiope's [http://cufp.galois.com/2008/slides/WrightGregory.pdf CUFP talk].<br />
<br />
* [http://www.att.com AT&amp;T]<br />
<blockquote><br />
Haskell is being used in the Network Security division to automate processing of internet abuse complaints. Haskell has allowed us to easily meet very tight deadlines with reliable results.<br />
</blockquote><br />
<br />
* [http://www.baml.com/ Bank of America Merril Lynch]<br />
<blockquote>Haskell is being used for backend data transformation and loading.</blockquote><br />
<br />
* [http://www.haskell.org/communities/12-2007/html/report.html#sect7.1.2 Barclays Capital Quantitative Analytics Group]<br />
<blockquote><br />
Barclays Capital's Quantitative Analytics group is using Haskell to<br />
develop an embedded domain-specific functional language (called FPF)<br />
which is used to specify exotic equity derivatives. These derivatives,<br />
which are naturally best described in terms of mathematical functions,<br />
and constructed compositionally, map well to being expressed in an<br />
embedded functional language. This language is now regularly being used<br />
by people who had no previous functional language experience.<br />
</blockquote><br />
::[http://lambda-the-ultimate.org/node/3331 Simon Frankau et al's JFP paper on their use of Haskell]<br />
<br />
* [http://bazqux.com BazQux Reader]<br />
<blockquote><br />
BazQux Reader is a commercial RSS reader. Its feeds and comments crawler and a part of web-server are implemented in Haskell.<br />
</blockquote><br />
<br />
* [http://www.bcode.com/ bCODE Pty Ltd] Sydney Australia<br />
<blockquote><br />
bCode Pty Ltd is a small venture capital-funded startup using Ocaml and a bit of Haskell in Sydney Australia.<br />
</blockquote><br />
<br />
* [http://www.bluespec.com/ Bluespec, Inc.] Waltham, Massachusetts<br />
<br />
<blockquote><br />
Developing a modern integrated circuit (ASIC or FPGA) is an enormously<br />
expensive process involving specification, modeling (to choose and fix the<br />
architecture), design (to describe what will become silicon) and verification<br />
(to ensure that it meets the specs), all before actually committing anything to<br />
silicon (where the cost of a failure can be tens of millions of dollars).<br />
Bluespec, Inc. is a three year-old company that provides language facilities,<br />
methodologies, and tools for this purpose, within the framework of the IEEE<br />
standard languages SystemVerilog and SystemC, but borrowing ideas heavily from<br />
Term Rewriting Systems and functional programming languages like Haskell. In<br />
this talk, after a brief technical overview to set the context, we will<br />
describe our tactics and strategies, and the challenges we face, in introducing<br />
declarative programming ideas into this field, both externally (convincing<br />
customers about the value of these ideas) and internally (using Haskell for our<br />
tool implementation). <br />
</blockquote><br />
<br />
::Bluespec's [http://cufp.galois.com/2006/abstracts.html#RishiyurNikhil CUFP talk].<br />
<br />
* [http://bu.mp/ Bump]<br />
<blockquote><br />
Bump use a Haskell-based server, [http://github.com/jamwt/Angel Angel], for process supervisor for all their backend systems, and [http://devblog.bu.mp/haskell-at-bump for other infrastructure tasks].<br />
</blockquote><br />
::Haskell at [http://fpcomplete.com/downloads/bump-fp-complete-study.pdf Bump]<br />
<br />
* [http://www.circos.com Circos Brand Karma] Singapore<br />
<blockquote><br />
Brand Karma provides services to brand owners to measure online sentiments towards their brands.<br />
Haskell is used in building parts of the product, specifically for back-end job scheduling and brand matching.<br />
</blockquote><br />
<br />
* [http://www.credit-suisse.com/ Credit Suisse Global Modelling and Analytics Group] London, UK; New York City, New York<br />
<br />
<blockquote><br />
GMAG, the quantitative modelling group at Credit Suisse, has been using Haskell<br />
for various projects since the beginning of 2006, with the twin aims of<br />
improving the productivity of modellers and making it easier for other people<br />
within the bank to use GMAG models. Current projects include: Further work on<br />
tools for checking, manipulating and transforming spreadsheets; a<br />
domain-specific language embedded in Haskell for implementing reusable<br />
components that can be compiled into various target forms (see the video presentation: [http://www.londonhug.net/2008/08/11/video-paradise-a-dsel-for-derivatives-pricing/ Paradise, a DSEL for Derivatives Pricing]).<br />
</blockquote><br />
<br />
::Credit Suisse's [http://cufp.galois.com/2006/abstracts.html#HowardMansell CUFP talk].<br />
<br />
* [http://detexify.kirelabs.org/classify.html Detexify]<br />
<br />
<blockquote><br />
Detexify is an online handwriting recognition system, whose backend is written in Haskell. <br />
</blockquote><br />
<br />
* [http://www.db.com/ Deutsche Bank Equity Proprietary Trading, Directional Credit Trading]<br />
<br />
<blockquote><br />
The Directional Credit Trading group uses Haskell as the primary<br />
implementation language for all its software infrastructure.<br />
</blockquote><br />
<br />
::Deutsche Bank's [http://cufp.galois.com/2008/abstracts.html#PolakowJeff CUFP talk].<br />
<br />
* [http://article.gmane.org/gmane.comp.lang.haskell.cafe/37093 Eaton] Cleveland, Ohio<br />
<br />
<blockquote><br />
Design and verification of hydraulic hybrid vehicle systems<br />
</blockquote><br />
<br />
::Eaton's [http://cufp.galois.com/2008/abstracts.html#HawkinsTom CUFP talk]<br />
::Eaton's [http://www.haskell.org/pipermail/haskell-cafe/2009-April/060602.html experiences using a Haskell DSL]<br />
<br />
* [Ericsson AB]<br />
<blockquote><br />
Ericsson uses Haskell for the implementation of Feldspar, an EDSL for digital signal processing algorithms.<br />
</blockquote><br />
<br />
::Ericsson's [http://hackage.haskell.org/package/feldspar-compiler Feldspar compiler]<br />
<br />
* [http://facebook.com Facebook]<br />
<br />
<blockquote><br />
Facebook uses some Haskell internally for tools. [http://github.com/facebook/lex-pass/tree/master lex-pass] is a tool for programmatically manipulating a PHP code base via Haskell.<br />
</blockquote><br />
<br />
:: Facebook's [http://cufp.galois.com/2009/abstracts.html#ChristopherPiroEugeneLetuchy CUFP talk]<br />
<br />
* [http://www.factisresearch.com/ Factis Research]<br />
<blockquote><br />
Factis research, located in Freiburg, Germany, develops reliable and user-friendly mobile solutions. Our client software runs under J2ME, Symbian, iPhone OS, Android, and Blackberry. The server components are implemented in Python and Haskell. We are actively using Haskell for a number of projects, most of which are released under an open-source license.<br />
</blockquote><br />
<br />
:: Factis' [http://haskell.org/communities/05-2010/html/report.html#factisresearch HCAR submission]<br />
<br />
* [http://fortytools.com fortytools gmbh]<br />
<blockquote><br />
Located in Hamburg, Germany, we are developing web-based productivity tools for invoicing, customer management, resource scheduling and time tracking. While using Javascript for building rich frontend application in the browser, we use Haskell to implement the REST backends. Additionally, we do occasional project/client work as well.<br />
</blockquote><br />
<br />
:: Oh, and of course we develop and maintain [http://hayoo.info Hayoo!] :)<br />
<br />
* [http://www.funktional.info/index.php?id=7&L=1 Funktionale Programmierung Dr. Heinrich Hördegen], Munich, Germany<br />
<blockquote><br />
We develop software prototypes according to the Pareto principle: After spending only 20 percent of budget, we aim to provide already 80 percent of the software's functionality. We can realize this by constructing a 2080-software-prototype that we can further develop into a full-fledged solution...<br />
</blockquote><br />
<br />
* [http://www.galois.com/ Galois, Inc] Portland, Oregon<br />
<br />
<blockquote><br />
Galois designs and develops high confidence software for critical applications.<br />
Our innovative approach to software development provides high levels of<br />
assurance, yet its scalability enables us to address the most complex problems.<br />
We have successfully engineered projects under contract for corporations and<br />
government clients in the demanding application areas of security, information<br />
assurance and cryptography. <br />
</blockquote><br />
<br />
::Galois' [http://cufp.galois.com/2007/abstracts.html#JohnLaunchbury 2007 CUFP talk]<br />
::Galois' [http://cufp.org/conference/sessions/2011/theorem-based-derivation-aes-implementation 2011 CUFP talk]<br />
::Galois' [http://corp.galois.com/blog/2009/4/27/engineering-large-projects-in-haskell-a-decade-of-fp-at-galo.html retrospective on 10 years of industrial Haskell use]<br />
<br />
* [http://google.com Google]<br />
<br />
<blockquote><br />
Haskell is used on a small number of internal projects in Google, for internal IT infrastructure support. <br />
</blockquote><br />
<br />
::Google's [http://k1024.org/~iusty/papers/icfp10-haskell-reagent.pdf ICFP 2010 experience report on Haskell]<br />
<br />
* [http://glyde.com/ Glyde]<br />
<br />
<blockquote><br />
Glyde uses OCaml and Haskell for a few projects. Glyde uses Haskell for our client-side template source-to-source translator, which converts HAML-like view templates into JS code.<br />
</blockquote><br />
<br />
* [http://groupcommerce.com Group Commerce]<br />
<blockquote><br />
Group Commerce uses Haskell to drive the main component of their advertising infrastructure: a Snap Framework based web server. Haskell enabled quicker development, higher reliability, and better maintainability than other languages, without having to sacrifice performance.<br />
</blockquote><br />
<br />
* [http://humane-software.com Humane Software]<br />
<blockquote>We develop enterprise systems with de-coupled, asynchronous Haskell backends and Javascript UIs.<br><br />
For our current customer, an Internet connectivity provider, we wrote a solution for monitoring multiple remote machines and analyzing gigabytes of traffic samples. Haskell proved an excellent tool for the job. <br />
We were able to replace legacy systems in a granular, piece-by-piece manner, while delivering new features.</blockquote><br />
<br />
* [http://hustlerturf.com Hustler Turf Equipment] Hesston, Kansas<br />
<blockquote><br />
Designs, builds, and sells lawn mowers. We use quite a bit of Haskell, especially as a "glue language" for tying together data from different manufacturing-related systems. We also use it for some web apps that are deployed to our dealer network. There are also some uses for it doing sysadmin<br />
automation, such as adding/removing people from LDAP servers and the like<br />
</blockquote><br />
<br />
* [http://iba-cg.de/haskell.html iba Consulting Gesellschaft] - Intelligent business architecture for you. Leipzig, Germany<br />
<br />
<blockquote><br />
iba CG develops software for large companies: <br />
* risk analysis and reporting solution for power supply company; <br />
* contract management, assert management, booking and budgeting software for one of the worldwide leading accounting firm.<br />
</blockquote><br />
<br />
* [http://www.ics-ag.de/ Informatik Consulting Systems AG]<br />
<br />
<blockquote><br />
ICS AG developed a simulation and testing tool which based on a DSL (Domain Specific Language). The DSL is used for the description of architecture and behavior of distributed system components (event/message based, reactive). The compiler was written in Haskell (with target language Ada). The test system is used in some industrial projects.<br />
</blockquote><br />
<br />
* [http://ipwnstudios.com/ iPwn Studios]<br />
<blockquote><br />
ipwn studios is a video game studio founded in 2009 and based in the greater Boston area. They're developing a game engine in Haskell, and a number of games built on that engine, including an action-rpg for touchscreen devices called bloodknight.<br />
</blockquote><br />
<br />
* [http://www.ivu.de/uk/products/public-transport/ IVU Traffic Technologies AG]<br />
<blockquote><br />
The rostering group at IVU Traffic Technologies AG has been using Haskell to check rosters for compliance with EC regulations.<br />
<br />
Our implementation is based on an embedded DSL to combine the regulation’s single rules into a solver that not only decides on instances but, in the case of a faulty roster, finds an interpretation of the roster that is “favorable” in the sense that the error messages it entails are “helpful” in leading the dispatcher to the resolution of the issue at hand.<br />
<br />
The solver is both reliable (due to strong static typing and referential transparency — we have not experienced a failure in three years) and efficient (due to constraint propagation, a custom search strategy, and lazy evaluation).<br />
<br />
Our EC 561/2006 component is part of the IVU.crew software suite and as such is in wide-spread use all over Europe, both in planning and dispatch. So the next time you enter a regional bus, chances are that the driver’s roster was checked by Haskell.<br />
</blockquote><br />
<br />
* [http://www.janrain.com JanRain]<br />
<blockquote><br />
JanRain uses Haskell for network and web software. Read more about [http://www.janrain.com/blogs/haskell-janrain Haskell at JanRain] and in their [http://corp.galois.com/blog/2011/3/8/tech-talk-haskell-and-the-social-web.html tech talk at Galois]. JanRain's "[http://www.janrain.com/products/capture Capture]" user API product is built on Haskell's Snap webframework.<br />
</blockquote><br />
<br />
:: See Janrain's [http://corp.galois.com/blog/2011/4/22/tech-talk-video-haskell-and-the-social-web.html technical talk about their use of Snap]<br />
<br />
* [http://joyridelabs.de/game/ Joyride Laboratories]<br />
<br />
<blockquote><br />
Joyride Laboratories is an independent game development studio, founded in 2009 by Florian Hofer and Sönke Hahn. Their first game, "Nikki and the Robots" was released in 2011.<br />
</blockquote><br />
<br />
* [http://www.linspire.com/ Linspire]<br />
<br />
<blockquote><br />
Linspire, Inc. has used functional programming since its inception in 2001,<br />
beginning with extensive use of O'Caml, with a steady shift to Haskell as its<br />
implementations and libraries have matured. Hardware detection, software<br />
packaging and CGI web page generation are all areas where we have used<br />
functional programming extensively. Haskell's feature set lets us replace much<br />
of our use of little languages (e.g., bash or awk) and two-level languages (C<br />
or C++ bound to an interpreted language), allowing for faster development,<br />
better code sharing and ultimately faster implementations. Above all, we value<br />
static type checking for minimizing runtime errors in applications that run in<br />
unknown environments and for wrapping legacy programs in strongly typed<br />
functions to ensure that we pass valid arguments. <br />
</blockquote><br />
<br />
::Linspire's [http://cufp.galois.com/2006/abstracts.html#CliffordBeshers CUFP talk]<br />
::Linspire's experience report on using [http://portal.acm.org/citation.cfm?doid=1291151.1291184 functional programming to manage a Linux distribution]<br />
<br />
* [http://www.mitre.org/ MITRE]<br />
<blockquote><br />
MITRE uses Haskell for, amongst other things, the [http://hackage.haskell.org/package/cpsa analysis of cryptographic protocols].<br />
</blockquote><br />
<br />
* [http://ertos.nicta.com.au/research/sel4/ NICTA]<br />
<blockquote><br />
NICTA has used Haskell as part of a project to verify the L4 microkernel.<br />
</blockquote><br />
::[http://www.drdobbs.com/embedded/222400553 Read the Dr. Dobbs article on using Haskell and formal methods to verify a kernel]<br />
<br />
* [http://www.gb.nrao.edu NRAO]<br />
<blockquote><br />
NRAO has used Haskell to implement the core science algorithms for the Robert C. Byrd Green Bank Telescope (GBT) Dynamic Scheduling System ([http://www.gb.nrao.edu/dss DSS]).<br />
::Source code available on [https://github.com/nrao/antioch GitHub].<br />
</blockquote><br />
<br />
* [http://www.ns-sol.co.jp NS Solutions(NSSOL)] Tokyo, Japan<br />
<blockquote><br />
NS Solutions has employed Haskell since 2008 to develop its software<br />
packages including "BancMeasure", a mark-to-market accounting software<br />
package for financial institutions, "BancMeasure for IFRS" and<br />
"Mamecif", a data analysis package.<br />
"BancMeasure" and "Mamecif" are registered trademarks of NS Solutions Corporation in JAPAN.<br />
</blockquote><br />
<br />
* [http://www.nvidia.com/content/global/global.php NVIDIA]<br />
<blockquote><br />
At NVIDIA, we have a handful of in-house tools that are written in Haskell<br />
</blockquote><br />
<br />
* [http://blog.openomy.com/2008/01/case-study-using-haskell-and-happs-for.html Openomy]<br />
<br />
<blockquote><br />
Openomy's API v2.0 is developed in Haskell, using the<br />
[http://www.happs.org/ HAppS] web platform.<br />
</blockquote><br />
<br />
* [http://www.oblomov.com Oblomov]<br />
<br />
<blockquote><br />
Oblomov Systems is a one-person software company based in Utrecht, The Netherlands. Founded in 2009, Oblomov has since then been working on a number of Haskell-related projects. The main focus lies on web-applications and (web-based) editors. Haskell has turned out to be extremely useful for implementing web servers that communicate with JavaScript clients or iPhone apps.<br />
</blockquote><br />
<br />
:: [http://haskell.org/communities/05-2010/html/report.html#oblomov Oblomov's HCAR submission].<br />
<br />
* [http://www.patch-tag.com Patch-Tag: hosting for darcs]<br />
<blockquote><br />
Need somewhere to put your darcs code? Try us.<br />
<br />
Patch-Tag is built with [http://happstack.com happstack], the continuation of the project formerly known as HAppS.<br />
</blockquote><br />
<br />
* [http://www.peerium.com Peerium, Inc] Cambridge, Massachusetts<br />
<blockquote><br />
At Peerium, we're striving to bring a new level of quality and efficiency to online communication and collaboration within virtual communities, social networks, and business environments. We believe that a new environment that supports the effortless sharing of both information and software will enable a level of online cooperation far beyond current Web-based technologies -- modern programming techniques will enable the creation of more robust and more powerful programs within these environments. To this end, we're building a new software platform for direct, real-time communication and collaboration within graphically rich environments. Peerium is located in the heart of Harvard Square in Cambridge, Massachusetts.<br />
</blockquote><br />
<br />
* [http://www.qualcomm.com/ Qualcomm, Inc]<br />
<br />
<blockquote><br />
Qualcomm uses Haskell to generate Lua bindings to the BREW platform <br />
</blockquote><br />
<br />
* [http://www.renci.org/ Renaissaince Computing Institute], Chapel Hill, North Carolina<br />
<blockquote><br />
The Renaissance Computing Institute (RENCI), a multi-institutional organization, brings together multidisciplinary experts and advanced technological capabilities to address pressing research issues and to find solutions to complex problems that affect the quality of life in North Carolina, our nation and the world.<br />
<br />
Research scientists at RENCI have used Haskell for a number of projects, including [http://vis.renci.org/jeff/2009/08/26/open-sourcing-the-big-board/ The Big Board].<br />
</blockquote><br />
<br />
::RENCI's [http://cufp.galois.com/2009/abstracts.html#JeffersonHeard CUFP talk].<br />
<br />
* [https://scrive.com/gb/en Scrive] <br />
<br />
<blockquote><br />
Scrive is a service for e-signing tenders, contracts, and other documents. We help our clients close deals faster, decrease their administrative burden, and improve their customers’ experience.<br />
</blockquote><br />
<br />
* [http://sankelsoftware.com Sankel Software] Albuquerque, New Mexico<br />
<br />
<blockquote><br />
Sankel Software has been using Haskell since 2002 for both prototyping and deployment for technologies ranging from CAD/CAM to gaming and computer animation. We specialize in the development of user-friendly, large, long-term applications that solve difficult and conceptually intricate problems.<br />
</blockquote><br />
<br />
* [http://www.signalicorp.com/index.htm Signali] Portland, Oregon<br />
<br />
<blockquote><br />
Signali Corp is a new custom hardware design company. Our chief products<br />
are custom IP cores targeted for embedded DSP and cryptographic<br />
applications. Our specialty is the design and implementation of<br />
computationally intensive, complex algorithms. The interfaces to each<br />
core are modular and can be very efficiently modified for your specific<br />
application. System-level integration and validation is crucial and is<br />
the majority of investment in a product.<br />
</blockquote><br />
<br />
* [http://www.standardchartered.com/home/en/index.html Standard Chartered]<br />
<br />
<blockquote><br />
Standard Chartered has a large group using Haskell for all aspects of its wholesale banking business.<br />
</blockquote><br />
<br />
* [http://www.starling-software.com/en/index.html Starling Software] Tokyo, Japan<br />
<blockquote><br />
Starling Software are developing a commercial automated options trading system <br />
in Haskell, and are migrating other parts of their software suite to<br />
Haskell.<br />
</blockquote><br />
<br />
::Starling Software's [http://www.starling-software.com/misc/icfp-2009-cjs.pdf experience building real time trading systems in Haskell] <br />
<br />
* [http://www.tabula.com/ Tabula.com]<br />
<blockquote><br />
Tabula is a privately held fabless semiconductor company developing 3-D Programmable Logic Devices. Haskell is used for internal compiler toolchains related to hardware design.<br />
</blockquote><br />
<br />
* [http://tsurucapital.com Tsuru Capital] Tokyo, Japan<br />
<blockquote><br />
Tsuru Capital is operating an automated options trading system written in Haskell.<br />
</blockquote><br />
<br />
::[http://haskell.org/communities/05-2010/html/report.html#sect7.6 Tsuru Capital's HCAR submission]<br />
<br />
* [http://tupil.com/ Tupil] Utrecht, The Netherlands<br />
<br />
<blockquote><br />
Tupil is a Dutch company that built software for clients, written in Haskell. Tupil used Haskell for the speed in development and resulting software quality. The company is founded by Chris Eidhof and Eelco Lempsink. Currently they build iPhone/iPad applications in Objective-C.<br />
</blockquote><br />
<br />
:: Tupil's experience building [http://blog.tupil.com/building-commercial-haskell-applications/ commercial web apps in Haskell]<br />
<br />
* [http://typlab.com TypLAB] Amsterdam, The Netherlands<br />
<br />
<blockquote><br />
TypLAB investigates and develops new ways of creating and consuming online content. Their [http://www.silkapp.com/ Silk] application makes it easy to filter and visualize large amounts of information.<br />
</blockquote><br />
<br />
:: TypLAB's blog on [http://blog.typlab.com/2009/09/why-we-use-haskell/ why they use Haskell]<br />
:: A [http://thenextweb.com/eu/2011/04/28/filter-and-visualize-data-in-seconds-with-silk/ review of Silk]<br />
<br />
* [http://www.sensor-sense.nl Sensor Sense] Nijmegen, The Netherlands<br />
<br />
<blockquote><br />
Sensor Sense is offering high technology systems for gas measurements in the ''ppbv'' down to ''pptv'' range. We use Haskell for the embedded control software of our trace gas detectors.<br />
</blockquote><br />
<br />
If you're using Haskell commercially, please add your details here.<br />
<br />
== The Industrial Haskell Group ==<br />
<br />
The [http://industry.haskell.org/ Industrial Haskell Group (IHG)] is an organisation to support the needs of commercial users of the Haskell programming language. <br />
<br />
== Jobs and recruitment ==<br />
<br />
[[Jobs|Haskell jobs]] on the HaskellWiki.<br />
<br />
[http://www.haskellers.com/jobs Jobs at Haskellers.com].<br />
<br />
== Consultants ==<br />
<br />
[[Consultants]]<br />
<br />
== Commercial Users of Functional Programming Workshop ==<br />
<br />
[http://www.galois.com/cufp/ Commercial Users of Functional Programming]<br />
<br />
The goal of [http://www.galois.com/cufp/ CUFP] is to build a community<br />
for users of functional programming languages and technology, be they<br />
using functional languages in their professional lives, in an open<br />
source project (other than implementation of functional languages), as a<br />
hobby, or any combination thereof. In short: anyone who uses functional<br />
programming as a means, but not an end.<br />
<br />
[[Category:Community]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Haskell_in_industry&diff=46153Haskell in industry2012-06-22T19:55:40Z<p>DonStewart: </p>
<hr />
<div>__NOTOC__<br />
<br />
Haskell has a diverse range of use commercially, from aerospace and defense, to finance, to web startups, hardware design firms and a lawnmower manufacturer. This page collects resources on the industrial use of Haskell.<br />
<br />
* The main user conference for industrial Haskell use is CUFP - the [http://cufp.org/ Commercial Users of Functional Programming Workshop].<br />
* The [http://industry.haskell.org Industrial Haskell Group] supports commercial users.<br />
* [http://fpcomplete.com/ FPComplete] is dedicated to the widespread adoption of modern Functional Programming technology, with a current focus on the Haskell system. <br />
<br />
== Haskell in Industry ==<br />
<br />
Many companies have used Haskell for a range of projects, including:<br />
<br />
* [http://cufp.galois.com/2007/abstracts.html#CyrilSchmidt ABN AMRO] Amsterdam, The Netherlands<br />
<blockquote><br />
ABN AMRO is an international bank headquartered in Amsterdam. For its<br />
investment banking activities it needs to measure the counterparty risk<br />
on portfolios of financial derivatives. </blockquote><br />
::ABN AMRO's [http://cufp.galois.com/2007/abstracts.html#CyrilSchmidt CUFP talk].<br />
<br />
* [http://www.aetion.com/ Aetion Technologies LLC] Columbus, Ohio<br />
<blockquote><br />
Aetion is a defense contractor whose applications use artificial intelligence.<br />
Rapidly changing priorities make it important to minimize the code impact of<br />
changes, which suits Haskell well. Aetion has developed three main projects in<br />
Haskell, all successful. Haskell's concise code was perhaps most important for<br />
rewriting: it made it practicable to throw away old code occasionally. DSELs<br />
allowed the AI to be specified very declaratively. <br />
</blockquote><br />
::Aetion's [http://cufp.galois.com/2006/slides/GaryMorris.pdf CUFP talk].<br />
<br />
* Alcatel-Lucent<br />
<blockquote><br />
A consortium of groups, including Alcatel-Lucent, have used Haskell to prototype narrowband software radio systems, running in (soft) real-time.<br />
</blockquote><br />
::Alcatel-Lucent's [http://cufp.org/conference/sessions/2011/fourteen-days-haskell-real-time-programming-projec CUFP talk]<br />
<br />
* [http://www.allstontrading.com/ Allston Trading]<br />
<blockquote><br />
Headquartered in Chicago, Illinois, Allston Trading, LLC is a premier high frequency market maker in over 40 financial exchanges, in 20 countries, and in nearly every conceivable product class. Allston makes some use of Haskell for their trading infrastructure.<br />
</blockquote><br />
<br />
* [http://www.amgen.com/ Amgen] Thousand Oaks, California<br />
<blockquote><br />
Amgen is a human therapeutics company in the biotechnology industry. Amgen pioneered the development of novel products based on advances in recombinant DNA and molecular biology and launched the biotechnology industry’s first blockbuster medicines.<br />
<br />
Amgen uses Haskell;<br />
<br />
* To rapidly build software to implement mathematical models and other complex, mathematically oriented applications<br />
* Provide a more mathematically rigorous validation of software<br />
* To break developers out of their software development rut by giving them a new way to think about software.<br />
</blockquote><br />
::Amgen's [http://cufp.galois.com/2008/abstracts.html#BalabanDavid CUFP talk].<br />
<br />
* [http://www.ansemond.com/ Ansemond LLC]<br />
<blockquote><br />
"Find It! Keep It! is a Mac Web Browser that lets you keep the pages you<br />
visit in a database. A list of these pages is shown in the 'database<br />
view'. "<br />
</blockquote><br />
<br />
* [http://antiope.com/ Antiope] Fair Haven, New Jersey<br />
<blockquote><br />
Antiope Associates provides custom solutions for wireless communication<br />
and networking problems. Our team has expertise in all aspects of<br />
wireless system design, from the physical and protocol layers to complex<br />
networked applications. Antiope Associates's relies on a number of<br />
advanced techniques to ensure that the communication systems we design<br />
are reliable and free from error. We use custom simulation tools<br />
developed in Haskell, to model our hardware designs..<br />
</blockquote><br />
::Antiope's [http://cufp.galois.com/2008/slides/WrightGregory.pdf CUFP talk].<br />
<br />
* [http://www.att.com AT&amp;T]<br />
<blockquote><br />
Haskell is being used in the Network Security division to automate processing of internet abuse complaints. Haskell has allowed us to easily meet very tight deadlines with reliable results.<br />
</blockquote><br />
<br />
* [http://www.baml.com/ Bank of America Merril Lynch]<br />
<blockquote>Haskell is being used for backend data transformation and loading.</blockquote><br />
<br />
* [http://www.haskell.org/communities/12-2007/html/report.html#sect7.1.2 Barclays Capital Quantitative Analytics Group]<br />
<blockquote><br />
Barclays Capital's Quantitative Analytics group is using Haskell to<br />
develop an embedded domain-specific functional language (called FPF)<br />
which is used to specify exotic equity derivatives. These derivatives,<br />
which are naturally best described in terms of mathematical functions,<br />
and constructed compositionally, map well to being expressed in an<br />
embedded functional language. This language is now regularly being used<br />
by people who had no previous functional language experience.<br />
</blockquote><br />
::[http://lambda-the-ultimate.org/node/3331 Simon Frankau et al's JFP paper on their use of Haskell]<br />
<br />
* [http://www.bcode.com/ bCODE Pty Ltd] Sydney Australia<br />
<blockquote><br />
bCode Pty Ltd is a small venture capital-funded startup using Ocaml and a bit of Haskell in Sydney Australia.<br />
</blockquote><br />
<br />
* [http://www.bluespec.com/ Bluespec, Inc.] Waltham, Massachusetts<br />
<br />
<blockquote><br />
Developing a modern integrated circuit (ASIC or FPGA) is an enormously<br />
expensive process involving specification, modeling (to choose and fix the<br />
architecture), design (to describe what will become silicon) and verification<br />
(to ensure that it meets the specs), all before actually committing anything to<br />
silicon (where the cost of a failure can be tens of millions of dollars).<br />
Bluespec, Inc. is a three year-old company that provides language facilities,<br />
methodologies, and tools for this purpose, within the framework of the IEEE<br />
standard languages SystemVerilog and SystemC, but borrowing ideas heavily from<br />
Term Rewriting Systems and functional programming languages like Haskell. In<br />
this talk, after a brief technical overview to set the context, we will<br />
describe our tactics and strategies, and the challenges we face, in introducing<br />
declarative programming ideas into this field, both externally (convincing<br />
customers about the value of these ideas) and internally (using Haskell for our<br />
tool implementation). <br />
</blockquote><br />
<br />
::Bluespec's [http://cufp.galois.com/2006/abstracts.html#RishiyurNikhil CUFP talk].<br />
<br />
* [http://bu.mp/ Bump]<br />
<blockquote><br />
Bump use a Haskell-based server, [http://github.com/jamwt/Angel Angel], for process supervisor for all their backend systems, and [http://devblog.bu.mp/haskell-at-bump for other infrastructure tasks].<br />
</blockquote><br />
<br />
* [http://www.circos.com Circos Brand Karma] Singapore<br />
<blockquote><br />
Brand Karma provides services to brand owners to measure online sentiments towards their brands.<br />
Haskell is used in building parts of the product, specifically for back-end job scheduling and brand matching.<br />
</blockquote><br />
<br />
* [http://www.credit-suisse.com/ Credit Suisse Global Modelling and Analytics Group] London, UK; New York City, New York<br />
<br />
<blockquote><br />
GMAG, the quantitative modelling group at Credit Suisse, has been using Haskell<br />
for various projects since the beginning of 2006, with the twin aims of<br />
improving the productivity of modellers and making it easier for other people<br />
within the bank to use GMAG models. Current projects include: Further work on<br />
tools for checking, manipulating and transforming spreadsheets; a<br />
domain-specific language embedded in Haskell for implementing reusable<br />
components that can be compiled into various target forms (see the video presentation: [http://www.londonhug.net/2008/08/11/video-paradise-a-dsel-for-derivatives-pricing/ Paradise, a DSEL for Derivatives Pricing]).<br />
</blockquote><br />
<br />
::Credit Suisse's [http://cufp.galois.com/2006/abstracts.html#HowardMansell CUFP talk].<br />
<br />
* [http://detexify.kirelabs.org/classify.html Detexify]<br />
<br />
<blockquote><br />
Detexify is an online handwriting recognition system, whose backend is written in Haskell. <br />
</blockquote><br />
<br />
* [http://www.db.com/ Deutsche Bank Equity Proprietary Trading, Directional Credit Trading]<br />
<br />
<blockquote><br />
The Directional Credit Trading group uses Haskell as the primary<br />
implementation language for all its software infrastructure.<br />
</blockquote><br />
<br />
::Deutsche Bank's [http://cufp.galois.com/2008/abstracts.html#PolakowJeff CUFP talk].<br />
<br />
* [http://article.gmane.org/gmane.comp.lang.haskell.cafe/37093 Eaton] Cleveland, Ohio<br />
<br />
<blockquote><br />
Design and verification of hydraulic hybrid vehicle systems<br />
</blockquote><br />
<br />
::Eaton's [http://cufp.galois.com/2008/abstracts.html#HawkinsTom CUFP talk]<br />
::Eaton's [http://www.haskell.org/pipermail/haskell-cafe/2009-April/060602.html experiences using a Haskell DSL]<br />
<br />
* [Ericsson AB]<br />
<blockquote><br />
Ericsson uses Haskell for the implementation of Feldspar, an EDSL for digital signal processing algorithms.<br />
</blockquote><br />
<br />
::Ericsson's [http://hackage.haskell.org/package/feldspar-compiler Feldspar compiler]<br />
<br />
* [http://facebook.com Facebook]<br />
<br />
<blockquote><br />
Facebook uses some Haskell internally for tools. [http://github.com/facebook/lex-pass/tree/master lex-pass] is a tool for programmatically manipulating a PHP code base via Haskell.<br />
</blockquote><br />
<br />
:: Facebook's [http://cufp.galois.com/2009/abstracts.html#ChristopherPiroEugeneLetuchy CUFP talk]<br />
<br />
* [http://www.factisresearch.com/ Factis Research]<br />
<blockquote><br />
Factis research, located in Freiburg, Germany, develops reliable and user-friendly mobile solutions. Our client software runs under J2ME, Symbian, iPhone OS, Android, and Blackberry. The server components are implemented in Python and Haskell. We are actively using Haskell for a number of projects, most of which are released under an open-source license.<br />
</blockquote><br />
<br />
:: Factis' [http://haskell.org/communities/05-2010/html/report.html#factisresearch HCAR submission]<br />
<br />
* [http://www.funktional.info/index.php?id=7&L=1 Funktionale Programmierung Dr. Heinrich Hördegen], Munich, Germany<br />
<blockquote><br />
We develop software prototypes according to the Pareto principle: After spending only 20 percent of budget, we aim to provide already 80 percent of the software's functionality. We can realize this by constructing a 2080-software-prototype that we can further develop into a full-fledged solution...<br />
</blockquote><br />
<br />
* [http://www.haskell.org/pipermail/haskell-cafe/2008-September/047168.html Gamr7] http://gamr7.com/ Lyon, France<br />
<br />
<blockquote><br />
Gamr7 was a startup focused on procedural city generation for the game<br />
and simulation market that operated from 2008 until 2012. <br />
</blockquote><br />
<br />
* [http://www.galois.com/ Galois, Inc] Portland, Oregon<br />
<br />
<blockquote><br />
Galois designs and develops high confidence software for critical applications.<br />
Our innovative approach to software development provides high levels of<br />
assurance, yet its scalability enables us to address the most complex problems.<br />
We have successfully engineered projects under contract for corporations and<br />
government clients in the demanding application areas of security, information<br />
assurance and cryptography. <br />
</blockquote><br />
<br />
::Galois' [http://cufp.galois.com/2007/abstracts.html#JohnLaunchbury 2007 CUFP talk]<br />
::Galois' [http://cufp.org/conference/sessions/2011/theorem-based-derivation-aes-implementation 2011 CUFP talk]<br />
::Galois' [http://corp.galois.com/blog/2009/4/27/engineering-large-projects-in-haskell-a-decade-of-fp-at-galo.html retrospective on 10 years of industrial Haskell use]<br />
<br />
* [http://google.com Google]<br />
<br />
<blockquote><br />
Haskell is used on a small number of internal projects in Google, for internal IT infrastructure support. <br />
</blockquote><br />
<br />
::Google's [http://k1024.org/~iusty/papers/icfp10-haskell-reagent.pdf ICFP 2010 experience report on Haskell]<br />
<br />
* [http://glyde.com/ Glyde]<br />
<br />
<blockquote><br />
Glyde uses OCaml and Haskell for a few projects. Glyde uses Haskell for our client-side template source-to-source translator, which converts HAML-like view templates into JS code.<br />
</blockquote><br />
<br />
* [http://groupcommerce.com Group Commerce]<br />
<br />
<blockquote><br />
Group Commerce uses Haskell to drive the main component of their advertising infrastructure: a Snap Framework based web server. Haskell enabled quicker development, higher reliability, and better maintainability than other languages, without having to sacrifice performance.<br />
</blockquote><br />
<br />
* [http://hustlerturf.com Hustler Turf Equipment] Hesston, Kansas<br />
<blockquote><br />
Designs, builds, and sells lawn mowers. We use quite a bit of Haskell, especially as a "glue language" for tying together data from different manufacturing-related systems. We also use it for some web apps that are deployed to our dealer network. There are also some uses for it doing sysadmin<br />
automation, such as adding/removing people from LDAP servers and the like<br />
</blockquote><br />
<br />
* [http://iba-cg.de/haskell.html iba Consulting Gesellschaft] - Intelligent business architecture for you. Leipzig, Germany<br />
<br />
<blockquote><br />
iba CG develops software for large companies: <br />
* risk analysis and reporting solution for power supply company; <br />
* contract management, assert management, booking and budgeting software for one of the worldwide leading accounting firm.<br />
</blockquote><br />
<br />
* [http://www.ics-ag.de/ Informatik Consulting Systems AG]<br />
<br />
<blockquote><br />
ICS AG developed a simulation and testing tool which based on a DSL (Domain Specific Language). The DSL is used for the description of architecture and behavior of distributed system components (event/message based, reactive). The compiler was written in Haskell (with target language Ada). The test system is used in some industrial projects.<br />
</blockquote><br />
<br />
* [http://ipwnstudios.com/ iPwn Studios]<br />
<blockquote><br />
ipwn studios is a video game studio founded in 2009 and based in the greater Boston area. They're developing a game engine in Haskell, and a number of games built on that engine, including an action-rpg for touchscreen devices called bloodknight.<br />
</blockquote><br />
<br />
* [http://www.ivu.de/uk/products/public-transport/ IVU Traffic Technologies AG]<br />
<blockquote><br />
The rostering group at IVU Traffic Technologies AG has been using Haskell to check rosters for compliance with EC regulations.<br />
<br />
Our implementation is based on an embedded DSL to combine the regulation’s single rules into a solver that not only decides on instances but, in the case of a faulty roster, finds an interpretation of the roster that is “favorable” in the sense that the error messages it entails are “helpful” in leading the dispatcher to the resolution of the issue at hand.<br />
<br />
The solver is both reliable (due to strong static typing and referential transparency — we have not experienced a failure in three years) and efficient (due to constraint propagation, a custom search strategy, and lazy evaluation).<br />
<br />
Our EC 561/2006 component is part of the IVU.crew software suite and as such is in wide-spread use all over Europe, both in planning and dispatch. So the next time you enter a regional bus, chances are that the driver’s roster was checked by Haskell.<br />
</blockquote><br />
<br />
* [http://www.janrain.com JanRain]<br />
<blockquote><br />
JanRain uses Haskell for network and web software. Read more about [http://www.janrain.com/blogs/haskell-janrain Haskell at JanRain] and in their [http://corp.galois.com/blog/2011/3/8/tech-talk-haskell-and-the-social-web.html tech talk at Galois]. JanRain's "[http://www.janrain.com/products/capture Capture]" user API product is built on Haskell's Snap webframework.<br />
</blockquote><br />
<br />
:: See Janrain's [http://corp.galois.com/blog/2011/4/22/tech-talk-video-haskell-and-the-social-web.html technical talk about their use of Snap]<br />
<br />
* [http://joyridelabs.de/game/ Joyride Laboratories]<br />
<br />
<blockquote><br />
Joyride Laboratories is an independent game development studio, founded in 2009 by Florian Hofer and Sönke Hahn. Their first game, "Nikki and the Robots" was released in 2011.<br />
</blockquote><br />
<br />
* [http://www.linspire.com/ Linspire]<br />
<br />
<blockquote><br />
Linspire, Inc. has used functional programming since its inception in 2001,<br />
beginning with extensive use of O'Caml, with a steady shift to Haskell as its<br />
implementations and libraries have matured. Hardware detection, software<br />
packaging and CGI web page generation are all areas where we have used<br />
functional programming extensively. Haskell's feature set lets us replace much<br />
of our use of little languages (e.g., bash or awk) and two-level languages (C<br />
or C++ bound to an interpreted language), allowing for faster development,<br />
better code sharing and ultimately faster implementations. Above all, we value<br />
static type checking for minimizing runtime errors in applications that run in<br />
unknown environments and for wrapping legacy programs in strongly typed<br />
functions to ensure that we pass valid arguments. <br />
</blockquote><br />
<br />
::Linspire's [http://cufp.galois.com/2006/abstracts.html#CliffordBeshers CUFP talk]<br />
::Linspire's experience report on using [http://portal.acm.org/citation.cfm?doid=1291151.1291184 functional programming to manage a Linux distribution]<br />
<br />
* [http://www.mitre.org/ MITRE]<br />
<blockquote><br />
MITRE uses Haskell for, amongst other things, the [http://hackage.haskell.org/package/cpsa analysis of cryptographic protocols].<br />
</blockquote><br />
<br />
* [http://ertos.nicta.com.au/research/sel4/ NICTA]<br />
<blockquote><br />
NICTA has used Haskell as part of a project to verify the L4 microkernel.<br />
</blockquote><br />
::[http://www.drdobbs.com/embedded/222400553 Read the Dr. Dobbs article on using Haskell and formal methods to verify a kernel]<br />
<br />
* [http://www.gb.nrao.edu NRAO]<br />
<blockquote><br />
NRAO has used Haskell to implement the core science algorithms for the Robert C. Byrd Green Bank Telescope (GBT) Dynamic Scheduling System ([http://www.gb.nrao.edu/dss DSS]).<br />
::Source code available on [https://github.com/nrao/antioch GitHub].<br />
</blockquote><br />
<br />
* [http://blog.openomy.com/2008/01/case-study-using-haskell-and-happs-for.html Openomy]<br />
<br />
<blockquote><br />
Openomy's API v2.0 is developed in Haskell, using the<br />
[http://www.happs.org/ HAppS] web platform.<br />
</blockquote><br />
<br />
* [http://www.oblomov.com Oblomov]<br />
<br />
<blockquote><br />
Oblomov Systems is a one-person software company based in Utrecht, The Netherlands. Founded in 2009, Oblomov has since then been working on a number of Haskell-related projects. The main focus lies on web-applications and (web-based) editors. Haskell has turned out to be extremely useful for implementing web servers that communicate with JavaScript clients or iPhone apps.<br />
</blockquote><br />
<br />
:: [http://haskell.org/communities/05-2010/html/report.html#oblomov Oblomov's HCAR submission].<br />
<br />
* [http://www.patch-tag.com Patch-Tag: hosting for darcs]<br />
<blockquote><br />
Need somewhere to put your darcs code? Try us.<br />
<br />
Patch-Tag is built with [http://happstack.com happstack], the continuation of the project formerly known as HAppS.<br />
</blockquote><br />
<br />
* [http://www.peerium.com Peerium, Inc] Cambridge, Massachusetts<br />
<blockquote><br />
At Peerium, we're striving to bring a new level of quality and efficiency to online communication and collaboration within virtual communities, social networks, and business environments. We believe that a new environment that supports the effortless sharing of both information and software will enable a level of online cooperation far beyond current Web-based technologies -- modern programming techniques will enable the creation of more robust and more powerful programs within these environments. To this end, we're building a new software platform for direct, real-time communication and collaboration within graphically rich environments. Peerium is located in the heart of Harvard Square in Cambridge, Massachusetts.<br />
</blockquote><br />
<br />
* [http://www.qualcomm.com/ Qualcomm, Inc]<br />
<br />
<blockquote><br />
Qualcomm uses Haskell to generate Lua bindings to the BREW platform <br />
</blockquote><br />
<br />
* [http://www.renci.org/ Renaissaince Computing Institute], Chapel Hill, North Carolina<br />
<blockquote><br />
The Renaissance Computing Institute (RENCI), a multi-institutional organization, brings together multidisciplinary experts and advanced technological capabilities to address pressing research issues and to find solutions to complex problems that affect the quality of life in North Carolina, our nation and the world.<br />
<br />
Research scientists at RENCI have used Haskell for a number of projects, including [http://vis.renci.org/jeff/2009/08/26/open-sourcing-the-big-board/ The Big Board].<br />
</blockquote><br />
<br />
::RENCI's [http://cufp.galois.com/2009/abstracts.html#JeffersonHeard CUFP talk].<br />
<br />
* [https://scrive.com/gb/en Scrive] <br />
<br />
<blockquote><br />
Scrive is a service for e-signing tenders, contracts, and other documents. We help our clients close deals faster, decrease their administrative burden, and improve their customers’ experience.<br />
</blockquote><br />
<br />
* [http://sankelsoftware.com Sankel Software] Albuquerque, New Mexico<br />
<br />
<blockquote><br />
Sankel Software has been using Haskell since 2002 for both prototyping and deployment for technologies ranging from CAD/CAM to gaming and computer animation. We specialize in the development of user-friendly, large, long-term applications that solve difficult and conceptually intricate problems.<br />
</blockquote><br />
<br />
* [http://www.signalicorp.com/index.htm Signali] Portland, Oregon<br />
<br />
<blockquote><br />
Signali Corp is a new custom hardware design company. Our chief products<br />
are custom IP cores targeted for embedded DSP and cryptographic<br />
applications. Our specialty is the design and implementation of<br />
computationally intensive, complex algorithms. The interfaces to each<br />
core are modular and can be very efficiently modified for your specific<br />
application. System-level integration and validation is crucial and is<br />
the majority of investment in a product.<br />
</blockquote><br />
<br />
* [http://www.standardchartered.com/home/en/index.html Standard Chartered]<br />
<br />
<blockquote><br />
Standard Chartered has a large group using Haskell for all aspects of its wholesale banking business.<br />
</blockquote><br />
<br />
* [http://www.starling-software.com/en/index.html Starling Software] Tokyo, Japan<br />
<blockquote><br />
Starling Software are developing a commercial automated options trading system <br />
in Haskell, and are migrating other parts of their software suite to<br />
Haskell.<br />
</blockquote><br />
<br />
::Starling Software's [http://www.starling-software.com/misc/icfp-2009-cjs.pdf experience building real time trading systems in Haskell] <br />
<br />
* [http://www.tabula.com/ Tabula.com]<br />
<blockquote><br />
Tabula is a privately held fabless semiconductor company developing 3-D Programmable Logic Devices. Haskell is used for internal compiler toolchains related to hardware design.<br />
</blockquote><br />
<br />
* [http://tsurucapital.com Tsuru Capital] Tokyo, Japan<br />
<blockquote><br />
Tsuru Capital is operating an automated options trading system written in Haskell.<br />
</blockquote><br />
<br />
::[http://haskell.org/communities/05-2010/html/report.html#sect7.6 Tsuru Capital's HCAR submission]<br />
<br />
* [http://tupil.com/ Tupil] Utrecht, The Netherlands<br />
<br />
<blockquote><br />
Tupil is a Dutch company that built software for clients, written in Haskell. Tupil used Haskell for the speed in development and resulting software quality. The company is founded by Chris Eidhof and Eelco Lempsink. Currently they build iPhone/iPad applications in Objective-C.<br />
</blockquote><br />
<br />
:: Tupil's experience building [http://blog.tupil.com/building-commercial-haskell-applications/ commercial web apps in Haskell]<br />
<br />
* [http://typlab.com TypLAB] Amsterdam, The Netherlands<br />
<br />
<blockquote><br />
TypLAB investigates and develops new ways of creating and consuming online content. Their [http://www.silkapp.com/ Silk] application makes it easy to filter and visualize large amounts of information.<br />
</blockquote><br />
<br />
:: TypLAB's blog on [http://blog.typlab.com/2009/09/why-we-use-haskell/ why they use Haskell]<br />
:: A [http://thenextweb.com/eu/2011/04/28/filter-and-visualize-data-in-seconds-with-silk/ review of Silk]<br />
<br />
* [http://www.sensor-sense.nl Sensor Sense] Nijmegen, The Netherlands<br />
<br />
<blockquote><br />
Sensor Sense is offering high technology systems for gas measurements in the ''ppbv'' down to ''pptv'' range. We use Haskell for the embedded control software of our trace gas detectors.<br />
</blockquote><br />
<br />
* [http://www.ns-sol.co.jp NS Solutions(NSSOL)] Tokyo, Japan<br />
<blockquote><br />
NS Solutions has employed Haskell since 2008 to develop its software<br />
packages including "BancMeasure", a mark-to-market accounting software<br />
package for financial institutions, "BancMeasure for IFRS" and<br />
"Mamecif", a data analysis package.<br />
"BancMeasure" and "Mamecif" are registered trademarks of NS Solutions Corporation in JAPAN.<br />
</blockquote><br />
<br />
If you're using Haskell commercially, please add your details here.<br />
<br />
== The Industrial Haskell Group ==<br />
<br />
The [http://industry.haskell.org/ Industrial Haskell Group (IHG)] is an organisation to support the needs of commercial users of the Haskell programming language. <br />
<br />
== Jobs and recruitment ==<br />
<br />
[[Jobs|Haskell jobs]] on the HaskellWiki.<br />
<br />
[http://www.haskellers.com/jobs Jobs at Haskellers.com].<br />
<br />
== Consultants ==<br />
<br />
[[Consultants]]<br />
<br />
== Commercial Users of Functional Programming Workshop ==<br />
<br />
[http://www.galois.com/cufp/ Commercial Users of Functional Programming]<br />
<br />
The goal of [http://www.galois.com/cufp/ CUFP] is to build a community<br />
for users of functional programming languages and technology, be they<br />
using functional languages in their professional lives, in an open<br />
source project (other than implementation of functional languages), as a<br />
hobby, or any combination thereof. In short: anyone who uses functional<br />
programming as a means, but not an end.<br />
<br />
[[Category:Community]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Haskell_in_industry&diff=46152Haskell in industry2012-06-22T19:27:17Z<p>DonStewart: </p>
<hr />
<div>__NOTOC__<br />
<br />
Haskell has a diverse range of use commercially, from aerospace and defense, to finance, to web startups, hardware design firms and a lawnmower manufacturer. This page collects resources on the industrial use of Haskell.<br />
<br />
* The main user conference for industrial Haskell use is CUFP - the [http://cufp.org/ Commercial Users of Functional Programming Workshop].<br />
* The [http://industry.haskell.org Industrial Haskell Group] supports commercial users.<br />
* [http://fpcomplete.com/ FPComplete] is dedicated to the widespread adoption of modern Functional Programming technology, with a current focus on the Haskell system. <br />
<br />
== Haskell in Industry ==<br />
<br />
Many companies have used Haskell for a range of projects, including:<br />
<br />
* [http://cufp.galois.com/2007/abstracts.html#CyrilSchmidt ABN AMRO] Amsterdam, The Netherlands<br />
<blockquote><br />
ABN AMRO is an international bank headquartered in Amsterdam. For its<br />
investment banking activities it needs to measure the counterparty risk<br />
on portfolios of financial derivatives. </blockquote><br />
::ABN AMRO's [http://cufp.galois.com/2007/abstracts.html#CyrilSchmidt CUFP talk].<br />
<br />
* [http://www.aetion.com/ Aetion Technologies LLC] Columbus, Ohio<br />
<blockquote><br />
Aetion is a defense contractor whose applications use artificial intelligence.<br />
Rapidly changing priorities make it important to minimize the code impact of<br />
changes, which suits Haskell well. Aetion has developed three main projects in<br />
Haskell, all successful. Haskell's concise code was perhaps most important for<br />
rewriting: it made it practicable to throw away old code occasionally. DSELs<br />
allowed the AI to be specified very declaratively. <br />
</blockquote><br />
::Aetion's [http://cufp.galois.com/2006/slides/GaryMorris.pdf CUFP talk].<br />
<br />
* Alcatel-Lucent<br />
<blockquote><br />
A consortium of groups, including Alcatel-Lucent, have used Haskell to prototype narrowband software radio systems, running in (soft) real-time.<br />
</blockquote><br />
::Alcatel-Lucent's [http://cufp.org/conference/sessions/2011/fourteen-days-haskell-real-time-programming-projec CUFP talk]<br />
<br />
* [http://www.allstontrading.com/ Allston Trading]<br />
<blockquote><br />
Headquartered in Chicago, Illinois, Allston Trading, LLC is a premier high frequency market maker in over 40 financial exchanges, in 20 countries, and in nearly every conceivable product class. Allston makes some use of Haskell for their trading infrastructure.<br />
</blockquote><br />
<br />
* [http://www.amgen.com/ Amgen] Thousand Oaks, California<br />
<blockquote><br />
Amgen is a human therapeutics company in the biotechnology industry. Amgen pioneered the development of novel products based on advances in recombinant DNA and molecular biology and launched the biotechnology industry’s first blockbuster medicines.<br />
<br />
Amgen uses Haskell;<br />
<br />
* To rapidly build software to implement mathematical models and other complex, mathematically oriented applications<br />
* Provide a more mathematically rigorous validation of software<br />
* To break developers out of their software development rut by giving them a new way to think about software.<br />
</blockquote><br />
::Amgen's [http://cufp.galois.com/2008/abstracts.html#BalabanDavid CUFP talk].<br />
<br />
* [http://www.ansemond.com/ Ansemond LLC]<br />
<blockquote><br />
"Find It! Keep It! is a Mac Web Browser that lets you keep the pages you<br />
visit in a database. A list of these pages is shown in the 'database<br />
view'. "<br />
</blockquote><br />
<br />
* [http://antiope.com/ Antiope] Fair Haven, New Jersey<br />
<blockquote><br />
Antiope Associates provides custom solutions for wireless communication<br />
and networking problems. Our team has expertise in all aspects of<br />
wireless system design, from the physical and protocol layers to complex<br />
networked applications. Antiope Associates's relies on a number of<br />
advanced techniques to ensure that the communication systems we design<br />
are reliable and free from error. We use custom simulation tools<br />
developed in Haskell, to model our hardware designs..<br />
</blockquote><br />
::Antiope's [http://cufp.galois.com/2008/slides/WrightGregory.pdf CUFP talk].<br />
<br />
* [http://www.att.com AT&amp;T]<br />
<blockquote><br />
Haskell is being used in the Network Security division to automate processing of internet abuse complaints. Haskell has allowed us to easily meet very tight deadlines with reliable results.<br />
</blockquote><br />
<br />
* [http://www.baml.com/ Bank of America Merril Lynch]<br />
<blockquote>Haskell is being used for backend data transformation and loading.</blockquote><br />
<br />
* [http://www.haskell.org/communities/12-2007/html/report.html#sect7.1.2 Barclays Capital Quantitative Analytics Group]<br />
<blockquote><br />
Barclays Capital's Quantitative Analytics group is using Haskell to<br />
develop an embedded domain-specific functional language (called FPF)<br />
which is used to specify exotic equity derivatives. These derivatives,<br />
which are naturally best described in terms of mathematical functions,<br />
and constructed compositionally, map well to being expressed in an<br />
embedded functional language. This language is now regularly being used<br />
by people who had no previous functional language experience.<br />
</blockquote><br />
::[http://lambda-the-ultimate.org/node/3331 Simon Frankau et al's JFP paper on their use of Haskell]<br />
<br />
* [http://www.bcode.com/ bCODE Pty Ltd] Sydney Australia<br />
<blockquote><br />
bCode Pty Ltd is a small venture capital-funded startup using Ocaml and a bit of Haskell in Sydney Australia.<br />
</blockquote><br />
<br />
* [http://www.bluespec.com/ Bluespec, Inc.] Waltham, Massachusetts<br />
<br />
<blockquote><br />
Developing a modern integrated circuit (ASIC or FPGA) is an enormously<br />
expensive process involving specification, modeling (to choose and fix the<br />
architecture), design (to describe what will become silicon) and verification<br />
(to ensure that it meets the specs), all before actually committing anything to<br />
silicon (where the cost of a failure can be tens of millions of dollars).<br />
Bluespec, Inc. is a three year-old company that provides language facilities,<br />
methodologies, and tools for this purpose, within the framework of the IEEE<br />
standard languages SystemVerilog and SystemC, but borrowing ideas heavily from<br />
Term Rewriting Systems and functional programming languages like Haskell. In<br />
this talk, after a brief technical overview to set the context, we will<br />
describe our tactics and strategies, and the challenges we face, in introducing<br />
declarative programming ideas into this field, both externally (convincing<br />
customers about the value of these ideas) and internally (using Haskell for our<br />
tool implementation). <br />
</blockquote><br />
<br />
::Bluespec's [http://cufp.galois.com/2006/abstracts.html#RishiyurNikhil CUFP talk].<br />
<br />
* [http://bu.mp/ Bump]<br />
<blockquote><br />
Bump use a Haskell-based server, [http://github.com/jamwt/Angel Angel], for process supervisor for all their backend systems, and [http://devblog.bu.mp/haskell-at-bump for other infrastructure tasks].<br />
</blockquote><br />
<br />
* [http://www.circos.com Circos Brand Karma] Singapore<br />
<blockquote><br />
Brand Karma provides services to brand owners to measure online sentiments towards their brands.<br />
Haskell is used in building parts of the product, specifically for back-end job scheduling and brand matching.<br />
</blockquote><br />
<br />
* [http://www.credit-suisse.com/ Credit Suisse Global Modelling and Analytics Group] London, UK; New York City, New York<br />
<br />
<blockquote><br />
GMAG, the quantitative modelling group at Credit Suisse, has been using Haskell<br />
for various projects since the beginning of 2006, with the twin aims of<br />
improving the productivity of modellers and making it easier for other people<br />
within the bank to use GMAG models. Current projects include: Further work on<br />
tools for checking, manipulating and transforming spreadsheets; a<br />
domain-specific language embedded in Haskell for implementing reusable<br />
components that can be compiled into various target forms (see the video presentation: [http://www.londonhug.net/2008/08/11/video-paradise-a-dsel-for-derivatives-pricing/ Paradise, a DSEL for Derivatives Pricing]).<br />
</blockquote><br />
<br />
::Credit Suisse's [http://cufp.galois.com/2006/abstracts.html#HowardMansell CUFP talk].<br />
<br />
* [http://detexify.kirelabs.org/classify.html Detexify]<br />
<br />
<blockquote><br />
Detexify is an online handwriting recognition system, whose backend is written in Haskell. <br />
</blockquote><br />
<br />
* [http://www.db.com/ Deutsche Bank Equity Proprietary Trading, Directional Credit Trading]<br />
<br />
<blockquote><br />
The Directional Credit Trading group uses Haskell as the primary<br />
implementation language for all its software infrastructure.<br />
</blockquote><br />
<br />
::Deutsche Bank's [http://cufp.galois.com/2008/abstracts.html#PolakowJeff CUFP talk].<br />
<br />
* [http://article.gmane.org/gmane.comp.lang.haskell.cafe/37093 Eaton] Cleveland, Ohio<br />
<br />
<blockquote><br />
Design and verification of hydraulic hybrid vehicle systems<br />
</blockquote><br />
<br />
::Eaton's [http://cufp.galois.com/2008/abstracts.html#HawkinsTom CUFP talk]<br />
::Eaton's [http://www.haskell.org/pipermail/haskell-cafe/2009-April/060602.html experiences using a Haskell DSL]<br />
<br />
* [Ericsson AB]<br />
<blockquote><br />
Ericsson uses Haskell for the implementation of Feldspar, an EDSL for digital signal processing algorithms.<br />
</blockquote><br />
<br />
::Ericsson's [http://hackage.haskell.org/package/feldspar-compiler Feldspar compiler]<br />
<br />
* [http://facebook.com Facebook]<br />
<br />
<blockquote><br />
Facebook uses some Haskell internally for tools. [http://github.com/facebook/lex-pass/tree/master lex-pass] is a tool for programmatically manipulating a PHP code base via Haskell.<br />
</blockquote><br />
<br />
:: Facebook's [http://cufp.galois.com/2009/abstracts.html#ChristopherPiroEugeneLetuchy CUFP talk]<br />
<br />
* [http://www.factisresearch.com/ Factis Research]<br />
<blockquote><br />
Factis research, located in Freiburg, Germany, develops reliable and user-friendly mobile solutions. Our client software runs under J2ME, Symbian, iPhone OS, Android, and Blackberry. The server components are implemented in Python and Haskell. We are actively using Haskell for a number of projects, most of which are released under an open-source license.<br />
</blockquote><br />
<br />
:: Factis' [http://haskell.org/communities/05-2010/html/report.html#factisresearch HCAR submission]<br />
<br />
* [http://www.funktional.info/index.php?id=7&L=1 Funktionale Programmierung Dr. Heinrich Hördegen], Munich, Germany<br />
<blockquote><br />
We develop software prototypes according to the Pareto principle: After spending only 20 percent of budget, we aim to provide already 80 percent of the software's functionality. We can realize this by constructing a 2080-software-prototype that we can further develop into a full-fledged solution...<br />
</blockquote><br />
<br />
* [http://www.haskell.org/pipermail/haskell-cafe/2008-September/047168.html Gamr7] http://gamr7.com/ Lyon, France<br />
<br />
<blockquote><br />
Gamr7 is a startup focused on procedural city generation for the game<br />
and simulation market. <br />
</blockquote><br />
<br />
* [http://www.galois.com/ Galois, Inc] Portland, Oregon<br />
<br />
<blockquote><br />
Galois designs and develops high confidence software for critical applications.<br />
Our innovative approach to software development provides high levels of<br />
assurance, yet its scalability enables us to address the most complex problems.<br />
We have successfully engineered projects under contract for corporations and<br />
government clients in the demanding application areas of security, information<br />
assurance and cryptography. <br />
</blockquote><br />
<br />
::Galois' [http://cufp.galois.com/2007/abstracts.html#JohnLaunchbury 2007 CUFP talk]<br />
::Galois' [http://cufp.org/conference/sessions/2011/theorem-based-derivation-aes-implementation 2011 CUFP talk]<br />
::Galois' [http://corp.galois.com/blog/2009/4/27/engineering-large-projects-in-haskell-a-decade-of-fp-at-galo.html retrospective on 10 years of industrial Haskell use]<br />
<br />
* [http://google.com Google]<br />
<br />
<blockquote><br />
Haskell is used on a small number of internal projects in Google, for internal IT infrastructure support. <br />
</blockquote><br />
<br />
::Google's [http://k1024.org/~iusty/papers/icfp10-haskell-reagent.pdf ICFP 2010 experience report on Haskell]<br />
<br />
* [http://glyde.com/ Glyde]<br />
<br />
<blockquote><br />
Glyde uses OCaml and Haskell for a few projects. Glyde uses Haskell for our client-side template source-to-source translator, which converts HAML-like view templates into JS code.<br />
</blockquote><br />
<br />
* [http://groupcommerce.com Group Commerce]<br />
<br />
<blockquote><br />
Group Commerce uses Haskell to drive the main component of their advertising infrastructure: a Snap Framework based web server. Haskell enabled quicker development, higher reliability, and better maintainability than other languages, without having to sacrifice performance.<br />
</blockquote><br />
<br />
* [http://hustlerturf.com Hustler Turf Equipment] Hesston, Kansas<br />
<blockquote><br />
Designs, builds, and sells lawn mowers. We use quite a bit of Haskell, especially as a "glue language" for tying together data from different manufacturing-related systems. We also use it for some web apps that are deployed to our dealer network. There are also some uses for it doing sysadmin<br />
automation, such as adding/removing people from LDAP servers and the like<br />
</blockquote><br />
<br />
* [http://iba-cg.de/haskell.html iba Consulting Gesellschaft] - Intelligent business architecture for you. Leipzig, Germany<br />
<br />
<blockquote><br />
iba CG develops software for large companies: <br />
* risk analysis and reporting solution for power supply company; <br />
* contract management, assert management, booking and budgeting software for one of the worldwide leading accounting firm.<br />
</blockquote><br />
<br />
* [http://www.ics-ag.de/ Informatik Consulting Systems AG]<br />
<br />
<blockquote><br />
ICS AG developed a simulation and testing tool which based on a DSL (Domain Specific Language). The DSL is used for the description of architecture and behavior of distributed system components (event/message based, reactive). The compiler was written in Haskell (with target language Ada). The test system is used in some industrial projects.<br />
</blockquote><br />
<br />
* [http://ipwnstudios.com/ iPwn Studios]<br />
<blockquote><br />
ipwn studios is a video game studio founded in 2009 and based in the greater Boston area. They're developing a game engine in Haskell, and a number of games built on that engine, including an action-rpg for touchscreen devices called bloodknight.<br />
</blockquote><br />
<br />
* [http://www.ivu.de/uk/products/public-transport/ IVU Traffic Technologies AG]<br />
<blockquote><br />
The rostering group at IVU Traffic Technologies AG has been using Haskell to check rosters for compliance with EC regulations.<br />
<br />
Our implementation is based on an embedded DSL to combine the regulation’s single rules into a solver that not only decides on instances but, in the case of a faulty roster, finds an interpretation of the roster that is “favorable” in the sense that the error messages it entails are “helpful” in leading the dispatcher to the resolution of the issue at hand.<br />
<br />
The solver is both reliable (due to strong static typing and referential transparency — we have not experienced a failure in three years) and efficient (due to constraint propagation, a custom search strategy, and lazy evaluation).<br />
<br />
Our EC 561/2006 component is part of the IVU.crew software suite and as such is in wide-spread use all over Europe, both in planning and dispatch. So the next time you enter a regional bus, chances are that the driver’s roster was checked by Haskell.<br />
</blockquote><br />
<br />
* [http://www.janrain.com JanRain]<br />
<blockquote><br />
JanRain uses Haskell for network and web software. Read more about [http://www.janrain.com/blogs/haskell-janrain Haskell at JanRain] and in their [http://corp.galois.com/blog/2011/3/8/tech-talk-haskell-and-the-social-web.html tech talk at Galois]. JanRain's "[http://www.janrain.com/products/capture Capture]" user API product is built on Haskell's Snap webframework.<br />
</blockquote><br />
<br />
:: See Janrain's [http://corp.galois.com/blog/2011/4/22/tech-talk-video-haskell-and-the-social-web.html technical talk about their use of Snap]<br />
<br />
* [http://joyridelabs.de/game/ Joyride Laboratories]<br />
<br />
<blockquote><br />
Joyride Laboratories is an independent game development studio, founded in 2009 by Florian Hofer and Sönke Hahn. Their first game, "Nikki and the Robots" was released in 2011.<br />
</blockquote><br />
<br />
* [http://www.linspire.com/ Linspire]<br />
<br />
<blockquote><br />
Linspire, Inc. has used functional programming since its inception in 2001,<br />
beginning with extensive use of O'Caml, with a steady shift to Haskell as its<br />
implementations and libraries have matured. Hardware detection, software<br />
packaging and CGI web page generation are all areas where we have used<br />
functional programming extensively. Haskell's feature set lets us replace much<br />
of our use of little languages (e.g., bash or awk) and two-level languages (C<br />
or C++ bound to an interpreted language), allowing for faster development,<br />
better code sharing and ultimately faster implementations. Above all, we value<br />
static type checking for minimizing runtime errors in applications that run in<br />
unknown environments and for wrapping legacy programs in strongly typed<br />
functions to ensure that we pass valid arguments. <br />
</blockquote><br />
<br />
::Linspire's [http://cufp.galois.com/2006/abstracts.html#CliffordBeshers CUFP talk]<br />
::Linspire's experience report on using [http://portal.acm.org/citation.cfm?doid=1291151.1291184 functional programming to manage a Linux distribution]<br />
<br />
* [http://www.mitre.org/ MITRE]<br />
<blockquote><br />
MITRE uses Haskell for, amongst other things, the [http://hackage.haskell.org/package/cpsa analysis of cryptographic protocols].<br />
</blockquote><br />
<br />
* [http://ertos.nicta.com.au/research/sel4/ NICTA]<br />
<blockquote><br />
NICTA has used Haskell as part of a project to verify the L4 microkernel.<br />
</blockquote><br />
::[http://www.drdobbs.com/embedded/222400553 Read the Dr. Dobbs article on using Haskell and formal methods to verify a kernel]<br />
<br />
* [http://www.gb.nrao.edu NRAO]<br />
<blockquote><br />
NRAO has used Haskell to implement the core science algorithms for the Robert C. Byrd Green Bank Telescope (GBT) Dynamic Scheduling System ([http://www.gb.nrao.edu/dss DSS]).<br />
::Source code available on [https://github.com/nrao/antioch GitHub].<br />
</blockquote><br />
<br />
* [http://blog.openomy.com/2008/01/case-study-using-haskell-and-happs-for.html Openomy]<br />
<br />
<blockquote><br />
Openomy's API v2.0 is developed in Haskell, using the<br />
[http://www.happs.org/ HAppS] web platform.<br />
</blockquote><br />
<br />
* [http://www.oblomov.com Oblomov]<br />
<br />
<blockquote><br />
Oblomov Systems is a one-person software company based in Utrecht, The Netherlands. Founded in 2009, Oblomov has since then been working on a number of Haskell-related projects. The main focus lies on web-applications and (web-based) editors. Haskell has turned out to be extremely useful for implementing web servers that communicate with JavaScript clients or iPhone apps.<br />
</blockquote><br />
<br />
:: [http://haskell.org/communities/05-2010/html/report.html#oblomov Oblomov's HCAR submission].<br />
<br />
* [http://www.patch-tag.com Patch-Tag: hosting for darcs]<br />
<blockquote><br />
Need somewhere to put your darcs code? Try us.<br />
<br />
Patch-Tag is built with [http://happstack.com happstack], the continuation of the project formerly known as HAppS.<br />
</blockquote><br />
<br />
* [http://www.peerium.com Peerium, Inc] Cambridge, Massachusetts<br />
<blockquote><br />
At Peerium, we're striving to bring a new level of quality and efficiency to online communication and collaboration within virtual communities, social networks, and business environments. We believe that a new environment that supports the effortless sharing of both information and software will enable a level of online cooperation far beyond current Web-based technologies -- modern programming techniques will enable the creation of more robust and more powerful programs within these environments. To this end, we're building a new software platform for direct, real-time communication and collaboration within graphically rich environments. Peerium is located in the heart of Harvard Square in Cambridge, Massachusetts.<br />
</blockquote><br />
<br />
* [http://www.qualcomm.com/ Qualcomm, Inc]<br />
<br />
<blockquote><br />
Qualcomm uses Haskell to generate Lua bindings to the BREW platform <br />
</blockquote><br />
<br />
* [http://www.renci.org/ Renaissaince Computing Institute], Chapel Hill, North Carolina<br />
<blockquote><br />
The Renaissance Computing Institute (RENCI), a multi-institutional organization, brings together multidisciplinary experts and advanced technological capabilities to address pressing research issues and to find solutions to complex problems that affect the quality of life in North Carolina, our nation and the world.<br />
<br />
Research scientists at RENCI have used Haskell for a number of projects, including [http://vis.renci.org/jeff/2009/08/26/open-sourcing-the-big-board/ The Big Board].<br />
</blockquote><br />
<br />
::RENCI's [http://cufp.galois.com/2009/abstracts.html#JeffersonHeard CUFP talk].<br />
<br />
* [https://scrive.com/gb/en Scrive] <br />
<br />
<blockquote><br />
Scrive is a service for e-signing tenders, contracts, and other documents. We help our clients close deals faster, decrease their administrative burden, and improve their customers’ experience.<br />
</blockquote><br />
<br />
* [http://sankelsoftware.com Sankel Software] Albuquerque, New Mexico<br />
<br />
<blockquote><br />
Sankel Software has been using Haskell since 2002 for both prototyping and deployment for technologies ranging from CAD/CAM to gaming and computer animation. We specialize in the development of user-friendly, large, long-term applications that solve difficult and conceptually intricate problems.<br />
</blockquote><br />
<br />
* [http://www.signalicorp.com/index.htm Signali] Portland, Oregon<br />
<br />
<blockquote><br />
Signali Corp is a new custom hardware design company. Our chief products<br />
are custom IP cores targeted for embedded DSP and cryptographic<br />
applications. Our specialty is the design and implementation of<br />
computationally intensive, complex algorithms. The interfaces to each<br />
core are modular and can be very efficiently modified for your specific<br />
application. System-level integration and validation is crucial and is<br />
the majority of investment in a product.<br />
</blockquote><br />
<br />
* [http://www.standardchartered.com/home/en/index.html Standard Chartered]<br />
<br />
<blockquote><br />
Standard Chartered has a large group using Haskell for all aspects of its wholesale banking business.<br />
</blockquote><br />
<br />
* [http://www.starling-software.com/en/index.html Starling Software] Tokyo, Japan<br />
<blockquote><br />
Starling Software are developing a commercial automated options trading system <br />
in Haskell, and are migrating other parts of their software suite to<br />
Haskell.<br />
</blockquote><br />
<br />
::Starling Software's [http://www.starling-software.com/misc/icfp-2009-cjs.pdf experience building real time trading systems in Haskell] <br />
<br />
* [http://www.tabula.com/ Tabula.com]<br />
<blockquote><br />
Tabula is a privately held fabless semiconductor company developing 3-D Programmable Logic Devices. Haskell is used for internal compiler toolchains related to hardware design.<br />
</blockquote><br />
<br />
* [http://tsurucapital.com Tsuru Capital] Tokyo, Japan<br />
<blockquote><br />
Tsuru Capital is operating an automated options trading system written in Haskell.<br />
</blockquote><br />
<br />
::[http://haskell.org/communities/05-2010/html/report.html#sect7.6 Tsuru Capital's HCAR submission]<br />
<br />
* [http://tupil.com/ Tupil] Utrecht, The Netherlands<br />
<br />
<blockquote><br />
Tupil is a Dutch company that built software for clients, written in Haskell. Tupil used Haskell for the speed in development and resulting software quality. The company is founded by Chris Eidhof and Eelco Lempsink. Currently they build iPhone/iPad applications in Objective-C.<br />
</blockquote><br />
<br />
:: Tupil's experience building [http://blog.tupil.com/building-commercial-haskell-applications/ commercial web apps in Haskell]<br />
<br />
* [http://typlab.com TypLAB] Amsterdam, The Netherlands<br />
<br />
<blockquote><br />
TypLAB investigates and develops new ways of creating and consuming online content. Their [http://www.silkapp.com/ Silk] application makes it easy to filter and visualize large amounts of information.<br />
</blockquote><br />
<br />
:: TypLAB's blog on [http://blog.typlab.com/2009/09/why-we-use-haskell/ why they use Haskell]<br />
:: A [http://thenextweb.com/eu/2011/04/28/filter-and-visualize-data-in-seconds-with-silk/ review of Silk]<br />
<br />
* [http://www.sensor-sense.nl Sensor Sense] Nijmegen, The Netherlands<br />
<br />
<blockquote><br />
Sensor Sense is offering high technology systems for gas measurements in the ''ppbv'' down to ''pptv'' range. We use Haskell for the embedded control software of our trace gas detectors.<br />
</blockquote><br />
<br />
* [http://www.ns-sol.co.jp NS Solutions(NSSOL)] Tokyo, Japan<br />
<blockquote><br />
NS Solutions has employed Haskell since 2008 to develop its software<br />
packages including "BancMeasure", a mark-to-market accounting software<br />
package for financial institutions, "BancMeasure for IFRS" and<br />
"Mamecif", a data analysis package.<br />
"BancMeasure" and "Mamecif" are registered trademarks of NS Solutions Corporation in JAPAN.<br />
</blockquote><br />
<br />
If you're using Haskell commercially, please add your details here.<br />
<br />
== The Industrial Haskell Group ==<br />
<br />
The [http://industry.haskell.org/ Industrial Haskell Group (IHG)] is an organisation to support the needs of commercial users of the Haskell programming language. <br />
<br />
== Jobs and recruitment ==<br />
<br />
[[Jobs|Haskell jobs]] on the HaskellWiki.<br />
<br />
[http://www.haskellers.com/jobs Jobs at Haskellers.com].<br />
<br />
== Consultants ==<br />
<br />
[[Consultants]]<br />
<br />
== Commercial Users of Functional Programming Workshop ==<br />
<br />
[http://www.galois.com/cufp/ Commercial Users of Functional Programming]<br />
<br />
The goal of [http://www.galois.com/cufp/ CUFP] is to build a community<br />
for users of functional programming languages and technology, be they<br />
using functional languages in their professional lives, in an open<br />
source project (other than implementation of functional languages), as a<br />
hobby, or any combination thereof. In short: anyone who uses functional<br />
programming as a means, but not an end.<br />
<br />
[[Category:Community]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Haskell_in_industry&diff=46149Haskell in industry2012-06-22T17:42:54Z<p>DonStewart: </p>
<hr />
<div>__NOTOC__<br />
<br />
Haskell has a diverse range of use commercially, from aerospace and defense, to finance, to web startups, hardware design firms and a lawnmower manufacturer. This page collects resources on the industrial use of Haskell.<br />
<br />
* The main user conference for industrial Haskell use is CUFP - the [http://cufp.org/ Commercial Users of Functional Programming Workshop].<br />
* The [http://industry.haskell.org Industrial Haskell Group] supports commercial users.<br />
* [http://fpcomplete.com/ FPComplete] is dedicated to the widespread adoption of modern Functional Programming technology, with a current focus on the Haskell system. <br />
<br />
== Haskell in Industry ==<br />
<br />
Many companies have used Haskell for a range of projects, including:<br />
<br />
* [http://cufp.galois.com/2007/abstracts.html#CyrilSchmidt ABN AMRO] Amsterdam, The Netherlands<br />
<blockquote><br />
ABN AMRO is an international bank headquartered in Amsterdam. For its<br />
investment banking activities it needs to measure the counterparty risk<br />
on portfolios of financial derivatives. </blockquote><br />
::ABN AMRO's [http://cufp.galois.com/2007/abstracts.html#CyrilSchmidt CUFP talk].<br />
<br />
* [http://www.aetion.com/ Aetion Technologies LLC] Columbus, Ohio<br />
<blockquote><br />
Aetion is a defense contractor whose applications use artificial intelligence.<br />
Rapidly changing priorities make it important to minimize the code impact of<br />
changes, which suits Haskell well. Aetion has developed three main projects in<br />
Haskell, all successful. Haskell's concise code was perhaps most important for<br />
rewriting: it made it practicable to throw away old code occasionally. DSELs<br />
allowed the AI to be specified very declaratively. <br />
</blockquote><br />
::Aetion's [http://cufp.galois.com/2006/slides/GaryMorris.pdf CUFP talk].<br />
<br />
* Alcatel-Lucent<br />
<blockquote><br />
A consortium of groups, including Alcatel-Lucent, have used Haskell to prototype narrowband software radio systems, running in (soft) real-time.<br />
</blockquote><br />
::Alcatel-Lucent's [http://cufp.org/conference/sessions/2011/fourteen-days-haskell-real-time-programming-projec CUFP talk]<br />
<br />
* [http://www.allstontrading.com/ Allston Trading]<br />
<blockquote><br />
Headquartered in Chicago, Illinois, Allston Trading, LLC is a premier high frequency market maker in over 40 financial exchanges, in 20 countries, and in nearly every conceivable product class. Allston makes some use of Haskell for their trading infrastructure.<br />
</blockquote><br />
<br />
* [http://www.amgen.com/ Amgen] Thousand Oaks, California<br />
<blockquote><br />
Amgen is a human therapeutics company in the biotechnology industry. Amgen pioneered the development of novel products based on advances in recombinant DNA and molecular biology and launched the biotechnology industry’s first blockbuster medicines.<br />
<br />
Amgen uses Haskell;<br />
<br />
* To rapidly build software to implement mathematical models and other complex, mathematically oriented applications<br />
* Provide a more mathematically rigorous validation of software<br />
* To break developers out of their software development rut by giving them a new way to think about software.<br />
</blockquote><br />
::Amgen's [http://cufp.galois.com/2008/abstracts.html#BalabanDavid CUFP talk].<br />
<br />
* [http://www.ansemond.com/ Ansemond LLC]<br />
<blockquote><br />
"Find It! Keep It! is a Mac Web Browser that lets you keep the pages you<br />
visit in a database. A list of these pages is shown in the 'database<br />
view'. "<br />
</blockquote><br />
<br />
* [http://antiope.com/ Antiope] Fair Haven, New Jersey<br />
<blockquote><br />
Antiope Associates provides custom solutions for wireless communication<br />
and networking problems. Our team has expertise in all aspects of<br />
wireless system design, from the physical and protocol layers to complex<br />
networked applications. Antiope Associates's relies on a number of<br />
advanced techniques to ensure that the communication systems we design<br />
are reliable and free from error. We use custom simulation tools<br />
developed in Haskell, to model our hardware designs..<br />
</blockquote><br />
::Antiope's [http://cufp.galois.com/2008/slides/WrightGregory.pdf CUFP talk].<br />
<br />
* [http://www.att.com AT&amp;T]<br />
<blockquote><br />
Haskell is being used in the Network Security division to automate processing of internet abuse complaints. Haskell has allowed us to easily meet very tight deadlines with reliable results.<br />
</blockquote><br />
<br />
* [http://www.baml.com/ Bank of America Merril Lynch]<br />
<blockquote>Haskell is being used for backend data transformation and loading.</blockquote><br />
<br />
* [http://www.haskell.org/communities/12-2007/html/report.html#sect7.1.2 Barclays Capital Quantitative Analytics Group]<br />
<blockquote><br />
Barclays Capital's Quantitative Analytics group is using Haskell to<br />
develop an embedded domain-specific functional language (called FPF)<br />
which is used to specify exotic equity derivatives. These derivatives,<br />
which are naturally best described in terms of mathematical functions,<br />
and constructed compositionally, map well to being expressed in an<br />
embedded functional language. This language is now regularly being used<br />
by people who had no previous functional language experience.<br />
</blockquote><br />
::[http://lambda-the-ultimate.org/node/3331 Simon Frankau et al's JFP paper on their use of Haskell]<br />
<br />
* [http://www.bcode.com/ bCODE Pty Ltd] Sydney Australia<br />
<blockquote><br />
bCode Pty Ltd is a small venture capital-funded startup using Ocaml and a bit of Haskell in Sydney Australia.<br />
</blockquote><br />
<br />
* [http://www.bluespec.com/ Bluespec, Inc.] Waltham, Massachusetts<br />
<br />
<blockquote><br />
Developing a modern integrated circuit (ASIC or FPGA) is an enormously<br />
expensive process involving specification, modeling (to choose and fix the<br />
architecture), design (to describe what will become silicon) and verification<br />
(to ensure that it meets the specs), all before actually committing anything to<br />
silicon (where the cost of a failure can be tens of millions of dollars).<br />
Bluespec, Inc. is a three year-old company that provides language facilities,<br />
methodologies, and tools for this purpose, within the framework of the IEEE<br />
standard languages SystemVerilog and SystemC, but borrowing ideas heavily from<br />
Term Rewriting Systems and functional programming languages like Haskell. In<br />
this talk, after a brief technical overview to set the context, we will<br />
describe our tactics and strategies, and the challenges we face, in introducing<br />
declarative programming ideas into this field, both externally (convincing<br />
customers about the value of these ideas) and internally (using Haskell for our<br />
tool implementation). <br />
</blockquote><br />
<br />
::Bluespec's [http://cufp.galois.com/2006/abstracts.html#RishiyurNikhil CUFP talk].<br />
<br />
* [http://bu.mp/ Bump]<br />
<blockquote><br />
Bump use a Haskell-based server, [http://github.com/jamwt/Angel Angel], for process supervisor for all their backend systems, and [http://devblog.bu.mp/haskell-at-bump for other infrastructure tasks].<br />
</blockquote><br />
<br />
* [http://www.circos.com Circos Brand Karma] Singapore<br />
<blockquote><br />
Brand Karma provides services to brand owners to measure online sentiments towards their brands.<br />
Haskell is used in building parts of the product, specifically for back-end job scheduling and brand matching.<br />
</blockquote><br />
<br />
* [http://www.credit-suisse.com/ Credit Suisse Global Modelling and Analytics Group] London, UK; New York City, New York<br />
<br />
<blockquote><br />
GMAG, the quantitative modelling group at Credit Suisse, has been using Haskell<br />
for various projects since the beginning of 2006, with the twin aims of<br />
improving the productivity of modellers and making it easier for other people<br />
within the bank to use GMAG models. Current projects include: Further work on<br />
tools for checking, manipulating and transforming spreadsheets; a<br />
domain-specific language embedded in Haskell for implementing reusable<br />
components that can be compiled into various target forms (see the video presentation: [http://www.londonhug.net/2008/08/11/video-paradise-a-dsel-for-derivatives-pricing/ Paradise, a DSEL for Derivatives Pricing]).<br />
</blockquote><br />
<br />
::Credit Suisse's [http://cufp.galois.com/2006/abstracts.html#HowardMansell CUFP talk].<br />
<br />
* [http://detexify.kirelabs.org/classify.html Detexify]<br />
<br />
<blockquote><br />
Detexify is an online handwriting recognition system, whose backend is written in Haskell. <br />
</blockquote><br />
<br />
* [http://www.db.com/ Deutsche Bank Equity Proprietary Trading, Directional Credit Trading]<br />
<br />
<blockquote><br />
The Directional Credit Trading group uses Haskell as the primary<br />
implementation language for all its software infrastructure.<br />
</blockquote><br />
<br />
::Deutsche Bank's [http://cufp.galois.com/2008/abstracts.html#PolakowJeff CUFP talk].<br />
<br />
* [http://article.gmane.org/gmane.comp.lang.haskell.cafe/37093 Eaton] Cleveland, Ohio<br />
<br />
<blockquote><br />
Design and verification of hydraulic hybrid vehicle systems<br />
</blockquote><br />
<br />
::Eaton's [http://cufp.galois.com/2008/abstracts.html#HawkinsTom CUFP talk]<br />
::Eaton's [http://www.haskell.org/pipermail/haskell-cafe/2009-April/060602.html experiences using a Haskell DSL]<br />
<br />
* [Ericsson AB]<br />
<blockquote><br />
Ericsson uses Haskell for the implementation of Feldspar, an EDSL for digital signal processing algorithms.<br />
</blockquote><br />
<br />
::Ericsson's [http://hackage.haskell.org/package/feldspar-compiler Feldspar compiler]<br />
<br />
* [http://facebook.com Facebook]<br />
<br />
<blockquote><br />
Facebook uses some Haskell internally for tools. [http://github.com/facebook/lex-pass/tree/master lex-pass] is a tool for programmatically manipulating a PHP code base via Haskell.<br />
</blockquote><br />
<br />
:: Facebook's [http://cufp.galois.com/2009/abstracts.html#ChristopherPiroEugeneLetuchy CUFP talk]<br />
<br />
* [http://www.factisresearch.com/ Factis Research]<br />
<blockquote><br />
Factis research, located in Freiburg, Germany, develops reliable and user-friendly mobile solutions. Our client software runs under J2ME, Symbian, iPhone OS, Android, and Blackberry. The server components are implemented in Python and Haskell. We are actively using Haskell for a number of projects, most of which are released under an open-source license.<br />
</blockquote><br />
<br />
:: Factis' [http://haskell.org/communities/05-2010/html/report.html#factisresearch HCAR submission]<br />
<br />
* [http://www.funktional.info/index.php?id=7&L=1 Funktionale Programmierung Dr. Heinrich Hördegen], Munich, Germany<br />
<blockquote><br />
We develop software prototypes according to the Pareto principle: After spending only 20 percent of budget, we aim to provide already 80 percent of the software's functionality. We can realize this by constructing a 2080-software-prototype that we can further develop into a full-fledged solution...<br />
</blockquote><br />
<br />
* [http://www.haskell.org/pipermail/haskell-cafe/2008-September/047168.html Gamr7] http://gamr7.com/ Lyon, France<br />
<br />
<blockquote><br />
Gamr7 is a startup focused on procedural city generation for the game<br />
and simulation market. <br />
</blockquote><br />
<br />
* [http://www.galois.com/ Galois, Inc] Portland, Oregon<br />
<br />
<blockquote><br />
Galois designs and develops high confidence software for critical applications.<br />
Our innovative approach to software development provides high levels of<br />
assurance, yet its scalability enables us to address the most complex problems.<br />
We have successfully engineered projects under contract for corporations and<br />
government clients in the demanding application areas of security, information<br />
assurance and cryptography. <br />
</blockquote><br />
<br />
::Galois' [http://cufp.galois.com/2007/abstracts.html#JohnLaunchbury 2007 CUFP talk]<br />
::Galois' [http://cufp.org/conference/sessions/2011/theorem-based-derivation-aes-implementation 2011 CUFP talk]<br />
::Galois' [http://corp.galois.com/blog/2009/4/27/engineering-large-projects-in-haskell-a-decade-of-fp-at-galo.html retrospective on 10 years of industrial Haskell use]<br />
<br />
* [http://google.com Google]<br />
<br />
<blockquote><br />
Haskell is used on a small number of internal projects in Google, for internal IT infrastructure support. <br />
</blockquote><br />
<br />
::Google's [http://k1024.org/~iusty/papers/icfp10-haskell-reagent.pdf ICFP 2010 experience report on Haskell]<br />
<br />
* [http://glyde.com/ Glyde]<br />
<br />
<blockquote><br />
Glyde uses OCaml and Haskell for a few projects. Glyde uses Haskell for our client-side template source-to-source translator, which converts HAML-like view templates into JS code.<br />
</blockquote><br />
<br />
* [http://article.gmane.org/gmane.comp.lang.haskell.cafe/21951 HAppS LLC]<br />
<br />
<blockquote><br />
Open web development company. Now merged with [http://happstack.com/ happstack]<br />
</blockquote><br />
<br />
* [http://hustlerturf.com Hustler Turf Equipment] Hesston, Kansas<br />
<blockquote><br />
Designs, builds, and sells lawn mowers. We use quite a bit of Haskell, especially as a "glue language" for tying together data from different manufacturing-related systems. We also use it for some web apps that are deployed to our dealer network. There are also some uses for it doing sysadmin<br />
automation, such as adding/removing people from LDAP servers and the like<br />
</blockquote><br />
<br />
* [http://iba-cg.de/haskell.html iba Consulting Gesellschaft] - Intelligent business architecture for you. Leipzig, Germany<br />
<br />
<blockquote><br />
iba CG develops software for large companies: <br />
* risk analysis and reporting solution for power supply company; <br />
* contract management, assert management, booking and budgeting software for one of the worldwide leading accounting firm.<br />
</blockquote><br />
<br />
* [http://www.ics-ag.de/ Informatik Consulting Systems AG]<br />
<br />
<blockquote><br />
ICS AG developed a simulation and testing tool which based on a DSL (Domain Specific Language). The DSL is used for the description of architecture and behavior of distributed system components (event/message based, reactive). The compiler was written in Haskell (with target language Ada). The test system is used in some industrial projects.<br />
</blockquote><br />
<br />
* [http://ipwnstudios.com/ iPwn Studios]<br />
<blockquote><br />
ipwn studios is a video game studio founded in 2009 and based in the greater Boston area. They're developing a game engine in Haskell, and a number of games built on that engine, including an action-rpg for touchscreen devices called bloodknight.<br />
</blockquote><br />
<br />
* [http://www.ivu.de/uk/products/public-transport/ IVU Traffic Technologies AG]<br />
<blockquote><br />
The rostering group at IVU Traffic Technologies AG has been using Haskell to check rosters for compliance with EC regulations.<br />
<br />
Our implementation is based on an embedded DSL to combine the regulation’s single rules into a solver that not only decides on instances but, in the case of a faulty roster, finds an interpretation of the roster that is “favorable” in the sense that the error messages it entails are “helpful” in leading the dispatcher to the resolution of the issue at hand.<br />
<br />
The solver is both reliable (due to strong static typing and referential transparency — we have not experienced a failure in three years) and efficient (due to constraint propagation, a custom search strategy, and lazy evaluation).<br />
<br />
Our EC 561/2006 component is part of the IVU.crew software suite and as such is in wide-spread use all over Europe, both in planning and dispatch. So the next time you enter a regional bus, chances are that the driver’s roster was checked by Haskell.<br />
</blockquote><br />
<br />
* [http://www.janrain.com JanRain]<br />
<blockquote><br />
JanRain uses Haskell for network and web software. Read more about [http://www.janrain.com/blogs/haskell-janrain Haskell at JanRain] and in their [http://corp.galois.com/blog/2011/3/8/tech-talk-haskell-and-the-social-web.html tech talk at Galois]. JanRain's "[http://www.janrain.com/products/capture Capture]" user API product is built on Haskell's Snap webframework.<br />
</blockquote><br />
<br />
:: See Janrain's [http://corp.galois.com/blog/2011/4/22/tech-talk-video-haskell-and-the-social-web.html technical talk about their use of Snap]<br />
<br />
* [http://joyridelabs.de/game/ Joyride Laboratories]<br />
<br />
<blockquote><br />
Joyride Laboratories is an independent game development studio, founded in 2009 by Florian Hofer and Sönke Hahn. Their first game, "Nikki and the Robots" was released in 2011.<br />
</blockquote><br />
<br />
* [http://www.linspire.com/ Linspire]<br />
<br />
<blockquote><br />
Linspire, Inc. has used functional programming since its inception in 2001,<br />
beginning with extensive use of O'Caml, with a steady shift to Haskell as its<br />
implementations and libraries have matured. Hardware detection, software<br />
packaging and CGI web page generation are all areas where we have used<br />
functional programming extensively. Haskell's feature set lets us replace much<br />
of our use of little languages (e.g., bash or awk) and two-level languages (C<br />
or C++ bound to an interpreted language), allowing for faster development,<br />
better code sharing and ultimately faster implementations. Above all, we value<br />
static type checking for minimizing runtime errors in applications that run in<br />
unknown environments and for wrapping legacy programs in strongly typed<br />
functions to ensure that we pass valid arguments. <br />
</blockquote><br />
<br />
::Linspire's [http://cufp.galois.com/2006/abstracts.html#CliffordBeshers CUFP talk]<br />
::Linspire's experience report on using [http://portal.acm.org/citation.cfm?doid=1291151.1291184 functional programming to manage a Linux distribution]<br />
<br />
* [http://www.mitre.org/ MITRE]<br />
<blockquote><br />
MITRE uses Haskell for, amongst other things, the [http://hackage.haskell.org/package/cpsa analysis of cryptographic protocols].<br />
</blockquote><br />
<br />
* [http://ertos.nicta.com.au/research/sel4/ NICTA]<br />
<blockquote><br />
NICTA has used Haskell as part of a project to verify the L4 microkernel.<br />
</blockquote><br />
::[http://www.drdobbs.com/embedded/222400553 Read the Dr. Dobbs article on using Haskell and formal methods to verify a kernel]<br />
<br />
* [http://www.gb.nrao.edu NRAO]<br />
<blockquote><br />
NRAO has used Haskell to implement the core science algorithms for the Robert C. Byrd Green Bank Telescope (GBT) Dynamic Scheduling System ([http://www.gb.nrao.edu/dss DSS]).<br />
::Source code available on [https://github.com/nrao/antioch GitHub].<br />
</blockquote><br />
<br />
* [http://blog.openomy.com/2008/01/case-study-using-haskell-and-happs-for.html Openomy]<br />
<br />
<blockquote><br />
Openomy's API v2.0 is developed in Haskell, using the<br />
[http://www.happs.org/ HAppS] web platform.<br />
</blockquote><br />
<br />
* [http://www.oblomov.com Oblomov]<br />
<br />
<blockquote><br />
Oblomov Systems is a one-person software company based in Utrecht, The Netherlands. Founded in 2009, Oblomov has since then been working on a number of Haskell-related projects. The main focus lies on web-applications and (web-based) editors. Haskell has turned out to be extremely useful for implementing web servers that communicate with JavaScript clients or iPhone apps.<br />
</blockquote><br />
<br />
:: [http://haskell.org/communities/05-2010/html/report.html#oblomov Oblomov's HCAR submission].<br />
<br />
* [http://www.patch-tag.com Patch-Tag: hosting for darcs]<br />
<blockquote><br />
Need somewhere to put your darcs code? Try us.<br />
<br />
Patch-Tag is built with [http://happstack.com happstack], the continuation of the project formerly known as HAppS.<br />
</blockquote><br />
<br />
* [http://www.peerium.com Peerium, Inc] Cambridge, Massachusetts<br />
<blockquote><br />
At Peerium, we're striving to bring a new level of quality and efficiency to online communication and collaboration within virtual communities, social networks, and business environments. We believe that a new environment that supports the effortless sharing of both information and software will enable a level of online cooperation far beyond current Web-based technologies -- modern programming techniques will enable the creation of more robust and more powerful programs within these environments. To this end, we're building a new software platform for direct, real-time communication and collaboration within graphically rich environments. Peerium is located in the heart of Harvard Square in Cambridge, Massachusetts.<br />
</blockquote><br />
<br />
* [http://www.qualcomm.com/ Qualcomm, Inc]<br />
<br />
<blockquote><br />
Qualcomm uses Haskell to generate Lua bindings to the BREW platform <br />
</blockquote><br />
<br />
* [http://www.renci.org/ Renaissaince Computing Institute], Chapel Hill, North Carolina<br />
<blockquote><br />
The Renaissance Computing Institute (RENCI), a multi-institutional organization, brings together multidisciplinary experts and advanced technological capabilities to address pressing research issues and to find solutions to complex problems that affect the quality of life in North Carolina, our nation and the world.<br />
<br />
Research scientists at RENCI have used Haskell for a number of projects, including [http://vis.renci.org/jeff/2009/08/26/open-sourcing-the-big-board/ The Big Board].<br />
</blockquote><br />
<br />
::RENCI's [http://cufp.galois.com/2009/abstracts.html#JeffersonHeard CUFP talk].<br />
<br />
* [https://scrive.com/gb/en Scrive] <br />
<br />
<blockquote><br />
Scrive is a service for e-signing tenders, contracts, and other documents. We help our clients close deals faster, decrease their administrative burden, and improve their customers’ experience.<br />
</blockquote><br />
<br />
* [http://sankelsoftware.com Sankel Software] Albuquerque, New Mexico<br />
<br />
<blockquote><br />
Sankel Software has been using Haskell since 2002 for both prototyping and deployment for technologies ranging from CAD/CAM to gaming and computer animation. We specialize in the development of user-friendly, large, long-term applications that solve difficult and conceptually intricate problems.<br />
</blockquote><br />
<br />
* [http://www.signalicorp.com/index.htm Signali] Portland, Oregon<br />
<br />
<blockquote><br />
Signali Corp is a new custom hardware design company. Our chief products<br />
are custom IP cores targeted for embedded DSP and cryptographic<br />
applications. Our specialty is the design and implementation of<br />
computationally intensive, complex algorithms. The interfaces to each<br />
core are modular and can be very efficiently modified for your specific<br />
application. System-level integration and validation is crucial and is<br />
the majority of investment in a product.<br />
</blockquote><br />
<br />
* [http://www.standardchartered.com/home/en/index.html Standard Chartered]<br />
<br />
<blockquote><br />
Standard Chartered has a large group using Haskell in finance.<br />
</blockquote><br />
<br />
* [http://www.starling-software.com/en/index.html Starling Software] Tokyo, Japan<br />
<blockquote><br />
Starling Software are developing a commercial automated options trading system <br />
in Haskell, and are migrating other parts of their software suite to<br />
Haskell.<br />
</blockquote><br />
<br />
::Starling Software's [http://www.starling-software.com/misc/icfp-2009-cjs.pdf experience building real time trading systems in Haskell] <br />
<br />
* [http://www.tabula.com/ Tabula.com]<br />
<blockquote><br />
Tabula is a privately held fabless semiconductor company developing 3-D Programmable Logic Devices. Haskell is used for internal compiler toolchains related to hardware design.<br />
</blockquote><br />
<br />
* [http://tsurucapital.com Tsuru Capital] Tokyo, Japan<br />
<blockquote><br />
Tsuru Capital is operating an automated options trading system written in Haskell.<br />
</blockquote><br />
<br />
::[http://haskell.org/communities/05-2010/html/report.html#sect7.6 Tsuru Capital's HCAR submission]<br />
<br />
* [http://tupil.com/ Tupil] Utrecht, The Netherlands<br />
<br />
<blockquote><br />
Tupil is a Dutch company that built software for clients, written in Haskell. Tupil used Haskell for the speed in development and resulting software quality. The company is founded by Chris Eidhof and Eelco Lempsink. Currently they build iPhone/iPad applications in Objective-C.<br />
</blockquote><br />
<br />
:: Tupil's experience building [http://blog.tupil.com/building-commercial-haskell-applications/ commercial web apps in Haskell]<br />
<br />
* [http://typlab.com TypLAB] Amsterdam, The Netherlands<br />
<br />
<blockquote><br />
TypLAB investigates and develops new ways of creating and consuming online content. Their [http://www.silkapp.com/ Silk] application makes it easy to filter and visualize large amounts of information.<br />
</blockquote><br />
<br />
:: TypLAB's blog on [http://blog.typlab.com/2009/09/why-we-use-haskell/ why they use Haskell]<br />
:: A [http://thenextweb.com/eu/2011/04/28/filter-and-visualize-data-in-seconds-with-silk/ review of Silk]<br />
<br />
* [http://www.sensor-sense.nl Sensor Sense] Nijmegen, The Netherlands<br />
<br />
<blockquote><br />
Sensor Sense is offering high technology systems for gas measurements in the ''ppbv'' down to ''pptv'' range. We use Haskell for the embedded control software of our trace gas detectors.<br />
</blockquote><br />
<br />
* [http://www.ns-sol.co.jp NS Solutions(NSSOL)] Tokyo, Japan<br />
<blockquote><br />
NS Solutions has employed Haskell since 2008 to develop its software<br />
packages including "BancMeasure", a mark-to-market accounting software<br />
package for financial institutions, "BancMeasure for IFRS" and<br />
"Mamecif", a data analysis package.<br />
"BancMeasure" and "Mamecif" are registered trademarks of NS Solutions Corporation in JAPAN.<br />
</blockquote><br />
<br />
If you're using Haskell commercially, please add your details here.<br />
<br />
== The Industrial Haskell Group ==<br />
<br />
The [http://industry.haskell.org/ Industrial Haskell Group (IHG)] is an organisation to support the needs of commercial users of the Haskell programming language. <br />
<br />
== Jobs and recruitment ==<br />
<br />
[[Jobs|Haskell jobs]] on the HaskellWiki.<br />
<br />
[http://www.haskellers.com/jobs Jobs at Haskellers.com].<br />
<br />
== Consultants ==<br />
<br />
[[Consultants]]<br />
<br />
== Commercial Users of Functional Programming Workshop ==<br />
<br />
[http://www.galois.com/cufp/ Commercial Users of Functional Programming]<br />
<br />
The goal of [http://www.galois.com/cufp/ CUFP] is to build a community<br />
for users of functional programming languages and technology, be they<br />
using functional languages in their professional lives, in an open<br />
source project (other than implementation of functional languages), as a<br />
hobby, or any combination thereof. In short: anyone who uses functional<br />
programming as a means, but not an end.<br />
<br />
[[Category:Community]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Haskell_in_industry&diff=46148Haskell in industry2012-06-22T17:41:18Z<p>DonStewart: </p>
<hr />
<div>__NOTOC__<br />
<br />
Haskell has a diverse range of use commercially, from aerospace and defense, to finance, to web startups, hardware design firms and a lawnmower manufacturer. This page collects resources on the industrial use of Haskell.<br />
<br />
* The main user conference for industrial Haskell use is CUFP - the [http://cufp.org/ Commercial Users of Functional Programming Workshop].<br />
* The [http://industry.haskell.org Industrial Haskell Group] supports commercial users.<br />
* [http://fpcomplete.com/ FPComplete] is dedicated to the widespread adoption of modern Functional Programming technology, with a current focus on the Haskell system. <br />
<br />
== Haskell in Industry ==<br />
<br />
Many companies have used Haskell for a range of projects, including:<br />
<br />
* [http://cufp.galois.com/2007/abstracts.html#CyrilSchmidt ABN AMRO] Amsterdam, The Netherlands<br />
<blockquote><br />
ABN AMRO is an international bank headquartered in Amsterdam. For its<br />
investment banking activities it needs to measure the counterparty risk<br />
on portfolios of financial derivatives. </blockquote><br />
::ABN AMRO's [http://cufp.galois.com/2007/abstracts.html#CyrilSchmidt CUFP talk].<br />
<br />
* [http://www.aetion.com/ Aetion Technologies LLC] Columbus, Ohio<br />
<blockquote><br />
Aetion is a defense contractor whose applications use artificial intelligence.<br />
Rapidly changing priorities make it important to minimize the code impact of<br />
changes, which suits Haskell well. Aetion has developed three main projects in<br />
Haskell, all successful. Haskell's concise code was perhaps most important for<br />
rewriting: it made it practicable to throw away old code occasionally. DSELs<br />
allowed the AI to be specified very declaratively. <br />
</blockquote><br />
::Aetion's [http://cufp.galois.com/2006/slides/GaryMorris.pdf CUFP talk].<br />
<br />
* Alcatel-Lucent<br />
<blockquote><br />
A consortium of groups, including Alcatel-Lucent, have used Haskell to prototype narrowband software radio sytems, running in (soft) real-time.<br />
</blockquote><br />
::Alcatel-Lucent's [http://cufp.org/conference/sessions/2011/fourteen-days-haskell-real-time-programming-projec CUFP talk]<br />
<br />
* [http://www.allstontrading.com/ Allston Trading]<br />
<blockquote><br />
Headquartered in Chicago, Illinois, Allston Trading, LLC is a premier high frequency market maker in over 40 financial exchanges, in 20 countries, and in nearly every conceivable product class. Allston makes some use of Haskell for their trading infrastructure.<br />
</blockquote><br />
<br />
* [http://www.amgen.com/ Amgen] Thousand Oaks, California<br />
<blockquote><br />
Amgen is a human therapeutics company in the biotechnology industry. Amgen pioneered the development of novel products based on advances in recombinant DNA and molecular biology and launched the biotechnology industry’s first blockbuster medicines.<br />
<br />
Amgen uses Haskell;<br />
<br />
* To rapidly build software to implement mathematical models and other complex, mathematically oriented applications<br />
* Provide a more mathematically rigorous validation of software<br />
* To break developers out of their software development rut by giving them a new way to think about software.<br />
</blockquote><br />
::Amgen's [http://cufp.galois.com/2008/abstracts.html#BalabanDavid CUFP talk].<br />
<br />
* [http://www.ansemond.com/ Ansemond LLC]<br />
<blockquote><br />
"Find It! Keep It! is a Mac Web Browser that lets you keep the pages you<br />
visit in a database. A list of these pages is shown in the 'database<br />
view'. "<br />
</blockquote><br />
<br />
* [http://antiope.com/ Antiope] Fair Haven, New Jersey<br />
<blockquote><br />
Antiope Associates provides custom solutions for wireless communication<br />
and networking problems. Our team has expertise in all aspects of<br />
wireless system design, from the physical and protocol layers to complex<br />
networked applications. Antiope Associates's relies on a number of<br />
advanced techniques to ensure that the communication systems we design<br />
are reliable and free from error. We use custom simulation tools<br />
developed in Haskell, to model our hardware designs..<br />
</blockquote><br />
::Antiope's [http://cufp.galois.com/2008/slides/WrightGregory.pdf CUFP talk].<br />
<br />
* [http://www.att.com AT&amp;T]<br />
<blockquote><br />
Haskell is being used in the Network Security division to automate processing of internet abuse complaints. Haskell has allowed us to easily meet very tight deadlines with reliable results.<br />
</blockquote><br />
<br />
* [http://www.baml.com/ Bank of America Merril Lynch]<br />
<blockquote>Haskell is being used for backend data transformation and loading.</blockquote><br />
<br />
* [http://www.haskell.org/communities/12-2007/html/report.html#sect7.1.2 Barclays Capital Quantitative Analytics Group]<br />
<blockquote><br />
Barclays Capital's Quantitative Analytics group is using Haskell to<br />
develop an embedded domain-specific functional language (called FPF)<br />
which is used to specify exotic equity derivatives. These derivatives,<br />
which are naturally best described in terms of mathematical functions,<br />
and constructed compositionally, map well to being expressed in an<br />
embedded functional language. This language is now regularly being used<br />
by people who had no previous functional language experience.<br />
</blockquote><br />
::[http://lambda-the-ultimate.org/node/3331 Simon Frankau et al's JFP paper on their use of Haskell]<br />
<br />
* [http://www.bcode.com/ bCODE Pty Ltd] Sydney Australia<br />
<blockquote><br />
bCode Pty Ltd is a small venture capital-funded startup using Ocaml and a bit of Haskell in Sydney Australia.<br />
</blockquote><br />
<br />
* [http://www.bluespec.com/ Bluespec, Inc.] Waltham, Massachusetts<br />
<br />
<blockquote><br />
Developing a modern integrated circuit (ASIC or FPGA) is an enormously<br />
expensive process involving specification, modeling (to choose and fix the<br />
architecture), design (to describe what will become silicon) and verification<br />
(to ensure that it meets the specs), all before actually committing anything to<br />
silicon (where the cost of a failure can be tens of millions of dollars).<br />
Bluespec, Inc. is a three year-old company that provides language facilities,<br />
methodologies, and tools for this purpose, within the framework of the IEEE<br />
standard languages SystemVerilog and SystemC, but borrowing ideas heavily from<br />
Term Rewriting Systems and functional programming languages like Haskell. In<br />
this talk, after a brief technical overview to set the context, we will<br />
describe our tactics and strategies, and the challenges we face, in introducing<br />
declarative programming ideas into this field, both externally (convincing<br />
customers about the value of these ideas) and internally (using Haskell for our<br />
tool implementation). <br />
</blockquote><br />
<br />
::Bluespec's [http://cufp.galois.com/2006/abstracts.html#RishiyurNikhil CUFP talk].<br />
<br />
* [http://bu.mp/ Bump]<br />
<blockquote><br />
Bump use a Haskell-based server, [http://github.com/jamwt/Angel Angel], for process supervisor for all their backend systems, and [http://devblog.bu.mp/haskell-at-bump for other infrastructure tasks].<br />
</blockquote><br />
<br />
* [http://www.circos.com Circos Brand Karma] Singapore<br />
<blockquote><br />
Brand Karma provides services to brand owners to measure online sentiments towards their brands.<br />
Haskell is used in building parts of the product, specifically for back-end job scheduling and brand matching.<br />
</blockquote><br />
<br />
* [http://www.credit-suisse.com/ Credit Suisse Global Modelling and Analytics Group] London, UK; New York City, New York<br />
<br />
<blockquote><br />
GMAG, the quantitative modelling group at Credit Suisse, has been using Haskell<br />
for various projects since the beginning of 2006, with the twin aims of<br />
improving the productivity of modellers and making it easier for other people<br />
within the bank to use GMAG models. Current projects include: Further work on<br />
tools for checking, manipulating and transforming spreadsheets; a<br />
domain-specific language embedded in Haskell for implementing reusable<br />
components that can be compiled into various target forms (see the video presentation: [http://www.londonhug.net/2008/08/11/video-paradise-a-dsel-for-derivatives-pricing/ Paradise, a DSEL for Derivatives Pricing]).<br />
</blockquote><br />
<br />
::Credit Suisse's [http://cufp.galois.com/2006/abstracts.html#HowardMansell CUFP talk].<br />
<br />
* [http://detexify.kirelabs.org/classify.html Detexify]<br />
<br />
<blockquote><br />
Detexify is an online handwriting recognition system, whose backend is written in Haskell. <br />
</blockquote><br />
<br />
* [http://www.db.com/ Deutsche Bank Equity Proprietary Trading, Directional Credit Trading]<br />
<br />
<blockquote><br />
The Directional Credit Trading group uses Haskell as the primary<br />
implementation language for all its software infrastructure.<br />
</blockquote><br />
<br />
::Deutsche Bank's [http://cufp.galois.com/2008/abstracts.html#PolakowJeff CUFP talk].<br />
<br />
* [http://article.gmane.org/gmane.comp.lang.haskell.cafe/37093 Eaton] Cleveland, Ohio<br />
<br />
<blockquote><br />
Design and verification of hydraulic hybrid vehicle systems<br />
</blockquote><br />
<br />
::Eaton's [http://cufp.galois.com/2008/abstracts.html#HawkinsTom CUFP talk]<br />
::Eaton's [http://www.haskell.org/pipermail/haskell-cafe/2009-April/060602.html experiences using a Haskell DSL]<br />
<br />
* [Ericsson AB]<br />
<blockquote><br />
Ericsson uses Haskell for the implementation of Feldspar, an EDSL for digital signal processing algorithms.<br />
</blockquote><br />
<br />
::Ericsson's [http://hackage.haskell.org/package/feldspar-compiler Feldspar compiler]<br />
<br />
* [http://facebook.com Facebook]<br />
<br />
<blockquote><br />
Facebook uses some Haskell internally for tools. [http://github.com/facebook/lex-pass/tree/master lex-pass] is a tool for programmatically manipulating a PHP code base via Haskell.<br />
</blockquote><br />
<br />
:: Facebook's [http://cufp.galois.com/2009/abstracts.html#ChristopherPiroEugeneLetuchy CUFP talk]<br />
<br />
* [http://www.factisresearch.com/ Factis Research]<br />
<blockquote><br />
Factis research, located in Freiburg, Germany, develops reliable and user-friendly mobile solutions. Our client software runs under J2ME, Symbian, iPhone OS, Android, and Blackberry. The server components are implemented in Python and Haskell. We are actively using Haskell for a number of projects, most of which are released under an open-source license.<br />
</blockquote><br />
<br />
:: Factis' [http://haskell.org/communities/05-2010/html/report.html#factisresearch HCAR submission]<br />
<br />
* [http://www.funktional.info/index.php?id=7&L=1 Funktionale Programmierung Dr. Heinrich Hördegen], Munich, Germany<br />
<blockquote><br />
We develop software prototypes according to the Pareto principle: After spending only 20 percent of budget, we aim to provide already 80 percent of the software's functionality. We can realize this by constructing a 2080-software-prototype that we can further develop into a full-fledged solution...<br />
</blockquote><br />
<br />
* [http://www.haskell.org/pipermail/haskell-cafe/2008-September/047168.html Gamr7] http://gamr7.com/ Lyon, France<br />
<br />
<blockquote><br />
Gamr7 is a startup focused on procedural city generation for the game<br />
and simulation market. <br />
</blockquote><br />
<br />
* [http://www.galois.com/ Galois, Inc] Portland, Oregon<br />
<br />
<blockquote><br />
Galois designs and develops high confidence software for critical applications.<br />
Our innovative approach to software development provides high levels of<br />
assurance, yet its scalability enables us to address the most complex problems.<br />
We have successfully engineered projects under contract for corporations and<br />
government clients in the demanding application areas of security, information<br />
assurance and cryptography. <br />
</blockquote><br />
<br />
::Galois' [http://cufp.galois.com/2007/abstracts.html#JohnLaunchbury 2007 CUFP talk]<br />
::Galois' [http://cufp.org/conference/sessions/2011/theorem-based-derivation-aes-implementation 2011 CUFP talk]<br />
::Galois' [http://corp.galois.com/blog/2009/4/27/engineering-large-projects-in-haskell-a-decade-of-fp-at-galo.html retrospective on 10 years of industrial Haskell use]<br />
<br />
* [http://google.com Google]<br />
<br />
<blockquote><br />
Haskell is used on a small number of internal projects in Google, for internal IT infrastructure support. <br />
</blockquote><br />
<br />
::Google's [http://k1024.org/~iusty/papers/icfp10-haskell-reagent.pdf ICFP 2010 experience report on Haskell]<br />
<br />
* [http://glyde.com/ Glyde]<br />
<br />
<blockquote><br />
Glyde uses OCaml and Haskell for a few projects. Glyde uses Haskell for our client-side template source-to-source translator, which converts HAML-like view templates into JS code.<br />
</blockquote><br />
<br />
* [http://article.gmane.org/gmane.comp.lang.haskell.cafe/21951 HAppS LLC]<br />
<br />
<blockquote><br />
Open web development company. Now merged with [http://happstack.com/ happstack]<br />
</blockquote><br />
<br />
* [http://hustlerturf.com Hustler Turf Equipment] Hesston, Kansas<br />
<blockquote><br />
Designs, builds, and sells lawn mowers. We use quite a bit of Haskell, especially as a "glue language" for tying together data from different manufacturing-related systems. We also use it for some web apps that are deployed to our dealer network. There are also some uses for it doing sysadmin<br />
automation, such as adding/removing people from LDAP servers and the like<br />
</blockquote><br />
<br />
* [http://iba-cg.de/haskell.html iba Consulting Gesellschaft] - Intelligent business architecture for you. Leipzig, Germany<br />
<br />
<blockquote><br />
iba CG develops software for large companies: <br />
* risk analysis and reporting solution for power supply company; <br />
* contract management, assert management, booking and budgeting software for one of the worldwide leading accounting firm.<br />
</blockquote><br />
<br />
* [http://www.ics-ag.de/ Informatik Consulting Systems AG]<br />
<br />
<blockquote><br />
ICS AG developed a simulation and testing tool which based on a DSL (Domain Specific Language). The DSL is used for the description of architecture and behavior of distributed system components (event/message based, reactive). The compiler was written in Haskell (with target language Ada). The test system is used in some industrial projects.<br />
</blockquote><br />
<br />
* [http://ipwnstudios.com/ iPwn Studios]<br />
<blockquote><br />
ipwn studios is a video game studio founded in 2009 and based in the greater Boston area. They're developing a game engine in Haskell, and a number of games built on that engine, including an action-rpg for touchscreen devices called bloodknight.<br />
</blockquote><br />
<br />
* [http://www.ivu.de/uk/products/public-transport/ IVU Traffic Technologies AG]<br />
<blockquote><br />
The rostering group at IVU Traffic Technologies AG has been using Haskell to check rosters for compliance with EC regulations.<br />
<br />
Our implementation is based on an embedded DSL to combine the regulation’s single rules into a solver that not only decides on instances but, in the case of a faulty roster, finds an interpretation of the roster that is “favorable” in the sense that the error messages it entails are “helpful” in leading the dispatcher to the resolution of the issue at hand.<br />
<br />
The solver is both reliable (due to strong static typing and referential transparency — we have not experienced a failure in three years) and efficient (due to constraint propagation, a custom search strategy, and lazy evaluation).<br />
<br />
Our EC 561/2006 component is part of the IVU.crew software suite and as such is in wide-spread use all over Europe, both in planning and dispatch. So the next time you enter a regional bus, chances are that the driver’s roster was checked by Haskell.<br />
</blockquote><br />
<br />
* [http://www.janrain.com JanRain]<br />
<blockquote><br />
JanRain uses Haskell for network and web software. Read more about [http://www.janrain.com/blogs/haskell-janrain Haskell at JanRain] and in their [http://corp.galois.com/blog/2011/3/8/tech-talk-haskell-and-the-social-web.html tech talk at Galois]. JanRain's "[http://www.janrain.com/products/capture Capture]" user API product is built on Haskell's Snap webframework.<br />
</blockquote><br />
<br />
:: See Janrain's [http://corp.galois.com/blog/2011/4/22/tech-talk-video-haskell-and-the-social-web.html technical talk about their use of Snap]<br />
<br />
* [http://joyridelabs.de/game/ Joyride Laboratories]<br />
<br />
<blockquote><br />
Joyride Laboratories is an independent game development studio, founded in 2009 by Florian Hofer and Sönke Hahn. Their first game, "Nikki and the Robots" was released in 2011.<br />
</blockquote><br />
<br />
* [http://www.linspire.com/ Linspire]<br />
<br />
<blockquote><br />
Linspire, Inc. has used functional programming since its inception in 2001,<br />
beginning with extensive use of O'Caml, with a steady shift to Haskell as its<br />
implementations and libraries have matured. Hardware detection, software<br />
packaging and CGI web page generation are all areas where we have used<br />
functional programming extensively. Haskell's feature set lets us replace much<br />
of our use of little languages (e.g., bash or awk) and two-level languages (C<br />
or C++ bound to an interpreted language), allowing for faster development,<br />
better code sharing and ultimately faster implementations. Above all, we value<br />
static type checking for minimizing runtime errors in applications that run in<br />
unknown environments and for wrapping legacy programs in strongly typed<br />
functions to ensure that we pass valid arguments. <br />
</blockquote><br />
<br />
::Linspire's [http://cufp.galois.com/2006/abstracts.html#CliffordBeshers CUFP talk]<br />
::Linspire's experience report on using [http://portal.acm.org/citation.cfm?doid=1291151.1291184 functional programming to manage a Linux distribution]<br />
<br />
* [http://www.mitre.org/ MITRE]<br />
<blockquote><br />
MITRE uses Haskell for, amongst other things, the [http://hackage.haskell.org/package/cpsa analysis of cryptographic protocols].<br />
</blockquote><br />
<br />
* [http://ertos.nicta.com.au/research/sel4/ NICTA]<br />
<blockquote><br />
NICTA has used Haskell as part of a project to verify the L4 microkernel.<br />
</blockquote><br />
::[http://www.drdobbs.com/embedded/222400553 Read the Dr. Dobbs article on using Haskell and formal methods to verify a kernel]<br />
<br />
* [http://www.gb.nrao.edu NRAO]<br />
<blockquote><br />
NRAO has used Haskell to implement the core science algorithms for the Robert C. Byrd Green Bank Telescope (GBT) Dynamic Scheduling System ([http://www.gb.nrao.edu/dss DSS]).<br />
::Source code available on [https://github.com/nrao/antioch GitHub].<br />
</blockquote><br />
<br />
* [http://blog.openomy.com/2008/01/case-study-using-haskell-and-happs-for.html Openomy]<br />
<br />
<blockquote><br />
Openomy's API v2.0 is developed in Haskell, using the<br />
[http://www.happs.org/ HAppS] web platform.<br />
</blockquote><br />
<br />
* [http://www.oblomov.com Oblomov]<br />
<br />
<blockquote><br />
Oblomov Systems is a one-person software company based in Utrecht, The Netherlands. Founded in 2009, Oblomov has since then been working on a number of Haskell-related projects. The main focus lies on web-applications and (web-based) editors. Haskell has turned out to be extremely useful for implementing web servers that communicate with JavaScript clients or iPhone apps.<br />
</blockquote><br />
<br />
:: [http://haskell.org/communities/05-2010/html/report.html#oblomov Oblomov's HCAR submission].<br />
<br />
* [http://www.patch-tag.com Patch-Tag: hosting for darcs]<br />
<blockquote><br />
Need somewhere to put your darcs code? Try us.<br />
<br />
Patch-Tag is built with [http://happstack.com happstack], the continuation of the project formerly known as HAppS.<br />
</blockquote><br />
<br />
* [http://www.peerium.com Peerium, Inc] Cambridge, Massachusetts<br />
<blockquote><br />
At Peerium, we're striving to bring a new level of quality and efficiency to online communication and collaboration within virtual communities, social networks, and business environments. We believe that a new environment that supports the effortless sharing of both information and software will enable a level of online cooperation far beyond current Web-based technologies -- modern programming techniques will enable the creation of more robust and more powerful programs within these environments. To this end, we're building a new software platform for direct, real-time communication and collaboration within graphically rich environments. Peerium is located in the heart of Harvard Square in Cambridge, Massachusetts.<br />
</blockquote><br />
<br />
* [http://www.qualcomm.com/ Qualcomm, Inc]<br />
<br />
<blockquote><br />
Qualcomm uses Haskell to generate Lua bindings to the BREW platform <br />
</blockquote><br />
<br />
* [http://www.renci.org/ Renaissaince Computing Institute], Chapel Hill, North Carolina<br />
<blockquote><br />
The Renaissance Computing Institute (RENCI), a multi-institutional organization, brings together multidisciplinary experts and advanced technological capabilities to address pressing research issues and to find solutions to complex problems that affect the quality of life in North Carolina, our nation and the world.<br />
<br />
Research scientists at RENCI have used Haskell for a number of projects, including [http://vis.renci.org/jeff/2009/08/26/open-sourcing-the-big-board/ The Big Board].<br />
</blockquote><br />
<br />
::RENCI's [http://cufp.galois.com/2009/abstracts.html#JeffersonHeard CUFP talk].<br />
<br />
* [https://scrive.com/gb/en Scrive] <br />
<br />
<blockquote><br />
Scrive is a service for e-signing tenders, contracts, and other documents. We help our clients close deals faster, decrease their administrative burden, and improve their customers’ experience.<br />
</blockquote><br />
<br />
* [http://sankelsoftware.com Sankel Software] Albuquerque, New Mexico<br />
<br />
<blockquote><br />
Sankel Software has been using Haskell since 2002 for both prototyping and deployment for technologies ranging from CAD/CAM to gaming and computer animation. We specialize in the development of user-friendly, large, long-term applications that solve difficult and conceptually intricate problems.<br />
</blockquote><br />
<br />
* [http://www.signalicorp.com/index.htm Signali] Portland, Oregon<br />
<br />
<blockquote><br />
Signali Corp is a new custom hardware design company. Our chief products<br />
are custom IP cores targeted for embedded DSP and cryptographic<br />
applications. Our specialty is the design and implementation of<br />
computationally intensive, complex algorithms. The interfaces to each<br />
core are modular and can be very efficiently modified for your specific<br />
application. System-level integration and validation is crucial and is<br />
the majority of investment in a product.<br />
</blockquote><br />
<br />
* [http://www.standardchartered.com/home/en/index.html Standard Chartered]<br />
<br />
<blockquote><br />
Standard Chartered has a large group using Haskell in finance.<br />
</blockquote><br />
<br />
* [http://www.starling-software.com/en/index.html Starling Software] Tokyo, Japan<br />
<blockquote><br />
Starling Software are developing a commercial automated options trading system <br />
in Haskell, and are migrating other parts of their software suite to<br />
Haskell.<br />
</blockquote><br />
<br />
::Starling Software's [http://www.starling-software.com/misc/icfp-2009-cjs.pdf experience building real time trading systems in Haskell] <br />
<br />
* [http://www.tabula.com/ Tabula.com]<br />
<blockquote><br />
Tabula is a privately held fabless semiconductor company developing 3-D Programmable Logic Devices. Haskell is used for internal compiler toolchains related to hardware design.<br />
</blockquote><br />
<br />
* [http://tsurucapital.com Tsuru Capital] Tokyo, Japan<br />
<blockquote><br />
Tsuru Capital is operating an automated options trading system written in Haskell.<br />
</blockquote><br />
<br />
::[http://haskell.org/communities/05-2010/html/report.html#sect7.6 Tsuru Capital's HCAR submission]<br />
<br />
* [http://tupil.com/ Tupil] Utrecht, The Netherlands<br />
<br />
<blockquote><br />
Tupil is a Dutch company that built software for clients, written in Haskell. Tupil used Haskell for the speed in development and resulting software quality. The company is founded by Chris Eidhof and Eelco Lempsink. Currently they build iPhone/iPad applications in Objective-C.<br />
</blockquote><br />
<br />
:: Tupil's experience building [http://blog.tupil.com/building-commercial-haskell-applications/ commercial web apps in Haskell]<br />
<br />
* [http://typlab.com TypLAB] Amsterdam, The Netherlands<br />
<br />
<blockquote><br />
TypLAB investigates and develops new ways of creating and consuming online content. Their [http://www.silkapp.com/ Silk] application makes it easy to filter and visualize large amounts of information.<br />
</blockquote><br />
<br />
:: TypLAB's blog on [http://blog.typlab.com/2009/09/why-we-use-haskell/ why they use Haskell]<br />
:: A [http://thenextweb.com/eu/2011/04/28/filter-and-visualize-data-in-seconds-with-silk/ review of Silk]<br />
<br />
* [http://www.sensor-sense.nl Sensor Sense] Nijmegen, The Netherlands<br />
<br />
<blockquote><br />
Sensor Sense is offering high technology systems for gas measurements in the ''ppbv'' down to ''pptv'' range. We use Haskell for the embedded control software of our trace gas detectors.<br />
</blockquote><br />
<br />
* [http://www.ns-sol.co.jp NS Solutions(NSSOL)] Tokyo, Japan<br />
<blockquote><br />
NS Solutions has employed Haskell since 2008 to develop its software<br />
packages including "BancMeasure", a mark-to-market accounting software<br />
package for financial institutions, "BancMeasure for IFRS" and<br />
"Mamecif", a data analysis package.<br />
"BancMeasure" and "Mamecif" are registered trademarks of NS Solutions Corporation in JAPAN.<br />
</blockquote><br />
<br />
If you're using Haskell commercially, please add your details here.<br />
<br />
== The Industrial Haskell Group ==<br />
<br />
The [http://industry.haskell.org/ Industrial Haskell Group (IHG)] is an organisation to support the needs of commercial users of the Haskell programming language. <br />
<br />
== Jobs and recruitment ==<br />
<br />
[[Jobs|Haskell jobs]] on the HaskellWiki.<br />
<br />
[http://www.haskellers.com/jobs Jobs at Haskellers.com].<br />
<br />
== Consultants ==<br />
<br />
[[Consultants]]<br />
<br />
== Commercial Users of Functional Programming Workshop ==<br />
<br />
[http://www.galois.com/cufp/ Commercial Users of Functional Programming]<br />
<br />
The goal of [http://www.galois.com/cufp/ CUFP] is to build a community<br />
for users of functional programming languages and technology, be they<br />
using functional languages in their professional lives, in an open<br />
source project (other than implementation of functional languages), as a<br />
hobby, or any combination thereof. In short: anyone who uses functional<br />
programming as a means, but not an end.<br />
<br />
[[Category:Community]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Wadler%27s_Law&diff=45283Wadler's Law2012-04-16T17:33:37Z<p>DonStewart: </p>
<hr />
<div>[[Category:Humor]]<br />
<br />
Wadler's Law states that:<br />
<br />
In any language design, the total time spent discussing<br />
a feature in this list is proportional to two raised to<br />
the power of its position.<br />
0. Semantics<br />
1. Syntax<br />
2. Lexical syntax<br />
3. Lexical syntax of comments<br />
<br />
The origins of this law are found in two emails:<br />
<br />
== Haskell mailing list, 1992 ==<br />
<br />
[http://www.cse.unsw.edu.au/~dons/haskell-1990-2006/msg00737.html Found here]<br />
<br />
Literate comments<br />
<br />
Date: Wed, 5 Feb 1992 09:14:00 +0000<br />
From: Philip Wadler <wadler@dcs.glasgow.ac.uk><br />
Sender: haskell-request@dcs.glasgow.ac.uk<br />
To: haskell@dcs.glasgow.ac.uk<br />
Cc: hudak@yale.edu, simonpj@dcs.glasgow.ac.uk, wadler@dcs.glasgow.ac.uk<br />
Subject: Literate comments<br />
<br />
Despite Paul having ruled on the matter, the debate still rages, with<br />
about half saying that > is wonderful, and half saying it is awful.<br />
<br />
Under these circumstances, Paul's ruling seems right: it is pointless<br />
to legislate > as part of the language.<br />
<br />
On the other hand, the current circumstance is that the main<br />
implementations do all use the > convention. It seems reasonable to<br />
say this in the report -- we should at least tell other implementors<br />
what the current implementors have settled upon. This would also let<br />
Joe use the convention in presenting the standard prelude, as he says<br />
he would like to do.<br />
<br />
To be precise: I propose an additional chapter of the report, labeled<br />
`Literate comments' and no more than one page long, that states a<br />
convention for providing literate comments, notes that it is NOT part<br />
of Haskell but is supported by existing implementations, and mentions<br />
that the Prelude is presented with this convention in the report. I<br />
volunteer to draft the page.<br />
<br />
Paul, as syntax honcho you should rule on this. Cheers, -- P<br />
<br />
PS: Wadler's Law:<br />
The emotional intensity of debate on a language feature<br />
increases as one moves down the following scale:<br />
Semantics,<br />
Syntax,<br />
Lexical syntax,<br />
Comments.<br />
<br />
== Revised law, Thu, 19 Dec 1996 ==<br />
<br />
[http://www.informatik.uni-kiel.de/~curry/listarchive/0017.html Found here]<br />
<br />
From: Philip Wadler <wadler_at_research.bell-labs.com><br />
Date: Thu, 19 Dec 1996 10:52:48 -0500<br />
<br />
> > Good point, and well worth stressing. Nonetheless, it still seems<br />
> > quite reasonable to say that Curry should contain all Haskell<br />
> > that eschew type classes and non-uniform pattern matching.<br />
><br />
> I think this is a good remark and I can agree to it.<br />
<br />
I'm delighted to hear this.<br />
<br />
> Concerning uppercase/lowercase: of course, we can or should<br />
> use Haskell's *convention*, but I do not like to enforce it, ...<br />
<br />
Oh dear. I don't think we've gotten our point across.<br />
<br />
Yes, there are problems with enforcing a capitalisation convention.<br />
There are also problems with not enforcing one. Nothing unique to<br />
Curry here. The Haskell committee spent a lot of time hashing<br />
over this ground. I strongly urge you to leave this can of worms<br />
shut. Adopt the Haskell solution wherever possible, warts and all!<br />
<br />
You might be interested in the following scientific law, which<br />
has been verified by extensive emperical observation.<br />
<br />
WADLER'S LAW OF LANGUAGE DESIGN<br />
<br />
In any language design, the total time spent discussing<br />
a feature in this list is proportional to two raised to<br />
the power of its position.<br />
<br />
0. Semantics<br />
1. Syntax<br />
2. Lexical syntax<br />
3. Lexical syntax of comments<br />
<br />
(That is, twice as much time is spent discussing syntax<br />
than semantics, twice as much time is spent discussing<br />
lexical syntax than syntax, and twice as much time is<br />
spent discussing syntax of comments than lexical syntax.)<br />
<br />
With the exception of comments, I would say discussion so far<br />
conforms to this observation. -- P</div>DonStewarthttps://wiki.haskell.org/index.php?title=Wadler%27s_Law&diff=45282Wadler's Law2012-04-16T17:33:00Z<p>DonStewart: </p>
<hr />
<div>[[Category:Humor]]<br />
<br />
Wadler's Law states that:<br />
<br />
In any language design, the total time spent discussing<br />
a feature is proportional to two raised to the power of its position.<br />
0. Semantics<br />
1. Syntax<br />
2. Lexical syntax<br />
3. Lexical syntax of comments<br />
<br />
The origins of this law are found in two emails:<br />
<br />
== Haskell mailing list, 1992 ==<br />
<br />
[http://www.cse.unsw.edu.au/~dons/haskell-1990-2006/msg00737.html Found here]<br />
<br />
Literate comments<br />
<br />
Date: Wed, 5 Feb 1992 09:14:00 +0000<br />
From: Philip Wadler <wadler@dcs.glasgow.ac.uk><br />
Sender: haskell-request@dcs.glasgow.ac.uk<br />
To: haskell@dcs.glasgow.ac.uk<br />
Cc: hudak@yale.edu, simonpj@dcs.glasgow.ac.uk, wadler@dcs.glasgow.ac.uk<br />
Subject: Literate comments<br />
<br />
Despite Paul having ruled on the matter, the debate still rages, with<br />
about half saying that > is wonderful, and half saying it is awful.<br />
<br />
Under these circumstances, Paul's ruling seems right: it is pointless<br />
to legislate > as part of the language.<br />
<br />
On the other hand, the current circumstance is that the main<br />
implementations do all use the > convention. It seems reasonable to<br />
say this in the report -- we should at least tell other implementors<br />
what the current implementors have settled upon. This would also let<br />
Joe use the convention in presenting the standard prelude, as he says<br />
he would like to do.<br />
<br />
To be precise: I propose an additional chapter of the report, labeled<br />
`Literate comments' and no more than one page long, that states a<br />
convention for providing literate comments, notes that it is NOT part<br />
of Haskell but is supported by existing implementations, and mentions<br />
that the Prelude is presented with this convention in the report. I<br />
volunteer to draft the page.<br />
<br />
Paul, as syntax honcho you should rule on this. Cheers, -- P<br />
<br />
PS: Wadler's Law:<br />
The emotional intensity of debate on a language feature<br />
increases as one moves down the following scale:<br />
Semantics,<br />
Syntax,<br />
Lexical syntax,<br />
Comments.<br />
<br />
== Revised law, Thu, 19 Dec 1996 ==<br />
<br />
[http://www.informatik.uni-kiel.de/~curry/listarchive/0017.html Found here]<br />
<br />
From: Philip Wadler <wadler_at_research.bell-labs.com><br />
Date: Thu, 19 Dec 1996 10:52:48 -0500<br />
<br />
> > Good point, and well worth stressing. Nonetheless, it still seems<br />
> > quite reasonable to say that Curry should contain all Haskell<br />
> > that eschew type classes and non-uniform pattern matching.<br />
><br />
> I think this is a good remark and I can agree to it.<br />
<br />
I'm delighted to hear this.<br />
<br />
> Concerning uppercase/lowercase: of course, we can or should<br />
> use Haskell's *convention*, but I do not like to enforce it, ...<br />
<br />
Oh dear. I don't think we've gotten our point across.<br />
<br />
Yes, there are problems with enforcing a capitalisation convention.<br />
There are also problems with not enforcing one. Nothing unique to<br />
Curry here. The Haskell committee spent a lot of time hashing<br />
over this ground. I strongly urge you to leave this can of worms<br />
shut. Adopt the Haskell solution wherever possible, warts and all!<br />
<br />
You might be interested in the following scientific law, which<br />
has been verified by extensive emperical observation.<br />
<br />
WADLER'S LAW OF LANGUAGE DESIGN<br />
<br />
In any language design, the total time spent discussing<br />
a feature in this list is proportional to two raised to<br />
the power of its position.<br />
<br />
0. Semantics<br />
1. Syntax<br />
2. Lexical syntax<br />
3. Lexical syntax of comments<br />
<br />
(That is, twice as much time is spent discussing syntax<br />
than semantics, twice as much time is spent discussing<br />
lexical syntax than syntax, and twice as much time is<br />
spent discussing syntax of comments than lexical syntax.)<br />
<br />
With the exception of comments, I would say discussion so far<br />
conforms to this observation. -- P</div>DonStewarthttps://wiki.haskell.org/index.php?title=IRC_channel&diff=43646IRC channel2011-12-17T21:44:02Z<p>DonStewart: </p>
<hr />
<div>Internet Relay Chat is a worldwide text chat service with many thousands<br />
of users among various irc networks.<br />
<br />
The Freenode IRC network hosts the very large #haskell channel, and we've had<br />
up to 884<br />
concurrent users<br />
(average is 787<br />
), making the channel consistently<br />
[http://searchirc.com/search.php?SCHANS=1&SSORT=SIZE&N=freenode one of the largest]<br />
of the thousands of channels on freenode. One famous<br />
resident is [[Lambdabot]], another is [http://hpaste.org hpaste] (see<br />
the [[#Bots|Bots]] section below).<br />
<br />
The IRC channel can be an excellent place to learn more about Haskell,<br />
and to just keep in the loop on new things in the Haskell world. Many<br />
new developments in the Haskell world first appear on the irc channel.<br />
<br />
Since 2009, the Haskell channel has grown large enough that we've split it in two parts:<br />
<br />
* #haskell, for all the usual things<br />
* #haskell-in-depth , for those seeking in depth, or more theoretical discussion<br />
<br />
As always, #haskell remains the primary place for new user questions.<br />
<br />
--------------------------<br />
<br />
[http://haskell.org/sitewiki/images/3/3c/Haskell-current.png [[Image:Haskell-current-small.png|thumb|The #haskell social graph, Jan 2008]]]<br />
<br />
[[Image:Irc-raw.png|thumb|Daily traffic in #haskell since 2004]]<br />
<br />
[[Image:Nick-activity.png|thumb|Growth of #haskell]]<br />
<br />
[[Image:Haskell-wordle-irc.png|thumb|Haskell noun map]]<br />
<br />
== Getting there ==<br />
<br />
If you point your irc client to [irc://chat.freenode.net/haskell chat.freenode.net] and then join the #haskell channel, you'll be there. Alternately, you can try http://java.freenode.net/ or http://webchat.freenode.net/ which connects inside the browser.<br />
<br />
Example, using [http://www.irssi.org/ irssi]:<br />
<br />
$ irssi -c chat.freenode.net -n myname -w mypassword<br />
/join #haskell<br />
<br />
Tip, if you're using Emacs to edit your Haskell sources then why not use it to chat about Haskell? Check out [http://www.emacswiki.org/cgi-bin/wiki/EmacsIRCClient ERC], The Emacs IRC client. Invoke it like this and follow the commands:<br />
<br />
M-x erc-select<br />
...<br />
/join #haskell<br />
<br />
[[Image:Irc--haskell-screenshot.png|frame|A screenshot of an irssi session in #haskell]]<br />
<br />
== Principles ==<br />
<br />
The #haskell channel is a very friendly, welcoming place to hang out,<br />
teach and learn. The goal of #haskell is to encourage learning and<br />
discussion of Haskell, functional programming, and programming in<br />
general. As part of this we welcome newbies, and encourage teaching of<br />
the language.<br />
<br />
Part of the #haskell success comes from the approach that the community<br />
is quite tight knit -- we know each other -- it's not just a homework<br />
channel. As a result, many collaborative projects have arisen between<br />
Haskell irc channel citizens.<br />
<br />
To maintain the friendly, open culture, the following is required:<br />
<br />
* Low to zero tolerance for ridiculing questions. Insulting new users is unacceptable.<br />
<br />
New Haskell users should feel entirely comfortable asking new questions.<br />
Helpful answers should be encouraged with <hask>name++</hask> karma<br />
points, in public, as a reward for providing a good answer.<br />
<br />
As the channel grows, we see a diverse range of people, with different<br />
programming backgrounds, trying to make their way with Haskell. A good<br />
rule of thumb, to avoid frustration is:<br />
<br />
* approach negative comments by asking for details (kind of like [http://en.wikipedia.org/wiki/Socratic_method Socratic questioning]), rather than challenging the competence of the writer (ad hominem).<br />
<br />
<br />
== History ==<br />
<br />
The #haskell channel appeared in the late 90s, and really got going<br />
in early 2001, with the help of Shae Erisson (aka shapr).<br />
<br />
A fairly extensive analysis of the traffic on #haskell over the years is<br />
[http://www.cse.unsw.edu.au/~dons/irc/ kept here]<br />
<br />
== Related channels ==<br />
<br />
In addition to the main Haskell channel there are also:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Channel<br />
! Purpose<br />
|-<br />
| #haskell-br<br />
| Brazilian Portuguese (pt_BR) speakers<br />
|-<br />
| #haskell.cz<br />
| Czech speakers (UTF-8)<br />
|- <br />
| #haskell.de<br />
| German speakers<br />
|-<br />
| #haskell.dut<br />
| Dutch speakers<br />
|-<br />
| #haskell.es<br />
| Spanish speakers<br />
|-<br />
| #haskell.fi<br />
| Finnish speakers<br />
|-<br />
| #haskell-fr (note the hyphen!)<br />
| French speakers <br />
|-<br />
| #haskell.hr<br />
| Croatian speakers<br />
|-<br />
| #haskell.it <br />
| Italian speakers<br />
|-<br />
| #haskell.jp <br />
| Japanese speakers<br />
|-<br />
| #haskell.no <br />
| Norwegian speakers<br />
|-<br />
| #haskell.pt<br />
| Portuguese speakers<br />
|-<br />
| #haskell.ru <br />
| Russian speakers. Seems that most of them migrated to Jabber conference (haskell@conference.jabber.ru).<br />
|-<br />
| #haskell_ru <br />
| Russian speakers again, in UTF-8. For those, who prefer good ol' IRC channel with a lambdabot.<br />
|-<br />
| #haskell.se <br />
| Swedish speakers<br />
|-<br />
| #haskell-blah <br />
| Haskell people talking about anything except Haskell itself<br />
|-<br />
| #haskell-books <br />
| Authors organizing the collaborative writing of the [http://en.wikibooks.org/wiki/Haskell Haskell wikibook] and other books or tutorials.<br />
|-<br />
| #haskell-game<br />
| The hub for Haskell-based [[Game Development|game development]]<br />
|-<br />
| #haskell-in-depth<br />
| slower paced discussion of use, theory, implementation etc with no monad tutorials!<br />
|-<br />
| #haskell-iphone<br />
| Haskell-based [[iPhone]] development<br />
|-<br />
| #haskell-overflow<br />
| Overflow conversations<br />
|-<br />
| #haskell-web<br />
| Friendly, practical discussion of haskell web app/framework/server development<br />
|-<br />
| '''Platform-specific:'''<br />
|<br />
|-<br />
| #arch-haskell <br />
| [[Arch Linux]]/ specific Haskell conversations<br />
|-<br />
| #gentoo-haskell <br />
| [[Gentoo]]/Linux specific Haskell conversations<br />
|-<br />
| '''Projects using haskell:'''<br />
|<br />
|-<br />
| #darcs <br />
| [[Darcs]] revision control system<br />
|-<br />
| #hackage<br />
| Haskell's software distribution infrastructure<br />
|-<br />
| #happs<br />
| [http://happstack.com Happstack] web framework<br />
|-<br />
| #hledger<br />
| [http://hledger.org hledger] accounting tools and library<br />
|-<br />
| #leksah<br />
| [http://leksah.org Leksah] IDE for Haskell development<br />
|-<br />
| #perl6 <br />
| [http://www.pugscode.org Perl 6] development (plenty of Haskell chat there too)<br />
|-<br />
| #snapframework<br />
| [http://snapframework.com/ Snap] web framework<br />
|-<br />
| #xmonad<br />
| [http://xmonad.org Xmonad] tiling window manager<br />
|}<br />
<br />
== Logs ==<br />
<br />
'''Logs''' are kept at http://tunes.org/~nef/logs/haskell/<br />
<br />
<!-- anywhere else? ircbrowse.com is a goner, apparently --><br />
<br />
== Bots ==<br />
<br />
There are various bots on the channel. Their names and usage are described here.<br />
<br />
=== lambdabot ===<br />
<br />
[[Lambdabot]] is both the name of a software package and a bot on the channel. It provides many useful services for visitors to the IRC channel. It is available as a haskell package and can be integrated into ghci. Details on the software are found on a [[Lambdabot|separate wiki page]].<br />
<br />
Here is its interface for the IRC user:<br />
<br />
lambdabot's commands are prepended by a '@' sign.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Command<br />
! Usage<br />
|-<br />
| @help<br />
| display help to other commands, but help text is not available for all commands.<br />
|-<br />
| @type EXPR or ':t' EXPR<br />
| shows the type of an expression<br />
|-<br />
| @kind TYPECONSTRUCTOR<br />
| shows the kind of a type constructor<br />
|-<br />
| @run EXPR or '>' EXPR<br />
| evaluates EXPR<br />
|-<br />
| @pl FUNCTION<br />
| shows a [[pointfree]] version of FUNCTION<br />
|-<br />
| @pointful FUNCTION or '@unpl' FUNCTION<br />
| shows a 'pointful' version of FUNCTION<br />
|}<br />
<br />
=== preflex ===<br />
<br />
is the name of a lambdabot with more commands/plugins enabled. It is run by ?? To talk to preflex, write <tt>preflex: command ARGS</tt><br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Command<br />
! Usage<br />
|-<br />
| help COMMAND<br />
| displays help to other commands.<br />
|-<br />
| list<br />
| lists all plugins with their commands<br />
|-<br />
| NICK++ / NICK--<br />
| in/decrements the karma of NICK.<br />
|-<br />
| karma NICK<br />
| shows the karma of NICK<br />
|-<br />
| seen NICK<br />
| shows information about the last message of a user<br />
|-<br />
| tell / ask<br />
| sends NICK MSG a message when she becomes active.<br />
|-<br />
| xseen<br />
| ''see 'seen' ?? any difference ?''<br />
|-<br />
| quote NICK<br />
| prints a random quote of NICK<br />
|-<br />
| remember NAME QUOTE<br />
| associates NAME with quote. can be accessed by 'quote'<br />
|-<br />
| ...<br />
| ...<br />
|}<br />
<br />
=== hpaste ===<br />
The hpaste bot provides a notification interface to the [http://hpaste.org hpaste pastebin]. [[Hpaste.el|Emacs integration]] is available.<br />
<br />
''Usage?''<br />
<br />
''Not online often !? ''<br />
<br />
=== hackage ===<br />
The hackage bot provides real-time notifications of new package uploads to [http://hackage.haskell.org Hackage].<br />
<br />
== Locations ==<br />
<br />
To get an overview of where everybody on the channel might<br />
be, physically, please visit [[Haskell user locations]].<br />
<br />
<br />
[[Category:Community]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Hackage_statistics&diff=42455Hackage statistics2011-10-16T17:39:55Z<p>DonStewart: </p>
<hr />
<div>A [http://www.jasani.org/2008/02/27/initial-release-of-hsparklines-010 sparkline] depicting statistics about the Haskell library repository, http://hackage.haskell.org. <br />
<br />
'''Updates this month by day'''<br />
<br />
http://i.imgur.com/a4yl9.png<br />
<br />
'''Updates per month since 2007'''<br />
<br />
http://i.imgur.com/wHuKG.png<br />
<br />
'''Releases of Haskell libraries and tools per day, on hackage, since inception'''<br />
<br />
http://i.imgur.com/4X5xP.png<br />
<br />
'''Available Haskell packages over time'''<br />
<br />
http://i.imgur.com/aAbsc.png<br />
<br />
(Via the Haskell [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/gnuplot gnuplot] bindings).<br />
<br />
'''Statistics'''<br />
<br />
http://hackage.haskell.org/cgi-bin/hackage-scripts/stats<br />
<br />
Graphics generated by the [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hsparklines hsparklines] library, by way of [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hackage-sparks-0.1 hackage-sparks].</div>DonStewarthttps://wiki.haskell.org/index.php?title=Hackage_statistics&diff=42454Hackage statistics2011-10-16T17:39:14Z<p>DonStewart: </p>
<hr />
<div>A [http://www.jasani.org/2008/02/27/initial-release-of-hsparklines-010 sparkline] depicting statistics about the Haskell library repository, http://hackage.haskell.org. <br />
<br />
'''Updates this month by day'''<br />
<br />
http://i.imgur.com/mHvNV.png<br />
<br />
'''Updates per month since 2007'''<br />
<br />
http://i.imgur.com/wHuKG.png<br />
<br />
'''Releases of Haskell libraries and tools per day, on hackage, since inception'''<br />
<br />
http://i.imgur.com/4X5xP.png<br />
<br />
'''Available Haskell packages over time'''<br />
<br />
http://i.imgur.com/aAbsc.png<br />
<br />
(Via the Haskell [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/gnuplot gnuplot] bindings).<br />
<br />
'''Statistics'''<br />
<br />
http://hackage.haskell.org/cgi-bin/hackage-scripts/stats<br />
<br />
Graphics generated by the [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hsparklines hsparklines] library, by way of [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hackage-sparks-0.1 hackage-sparks].</div>DonStewarthttps://wiki.haskell.org/index.php?title=Hackage_statistics&diff=42453Hackage statistics2011-10-16T17:38:20Z<p>DonStewart: </p>
<hr />
<div>A [http://www.jasani.org/2008/02/27/initial-release-of-hsparklines-010 sparkline] depicting statistics about the Haskell library repository, http://hackage.haskell.org. <br />
<br />
'''Updates this month by day'''<br />
<br />
http://i.imgur.com/mHvNV.png<br />
<br />
'''Updates per month since 2007'''<br />
<br />
http://i.imgur.com/YWYd7.png<br />
<br />
'''Releases of Haskell libraries and tools per day, on hackage, since inception'''<br />
<br />
http://i.imgur.com/4X5xP.png<br />
<br />
'''Available Haskell packages over time'''<br />
<br />
http://i.imgur.com/aAbsc.png<br />
<br />
(Via the Haskell [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/gnuplot gnuplot] bindings).<br />
<br />
'''Statistics'''<br />
<br />
http://hackage.haskell.org/cgi-bin/hackage-scripts/stats<br />
<br />
Graphics generated by the [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hsparklines hsparklines] library, by way of [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hackage-sparks-0.1 hackage-sparks].</div>DonStewarthttps://wiki.haskell.org/index.php?title=Hackage_statistics&diff=42452Hackage statistics2011-10-16T17:36:46Z<p>DonStewart: </p>
<hr />
<div>A [http://www.jasani.org/2008/02/27/initial-release-of-hsparklines-010 sparkline] depicting statistics about the Haskell library repository, http://hackage.haskell.org. <br />
<br />
'''Updates this month by day'''<br />
<br />
http://i.imgur.com/mHvNV.png<br />
<br />
'''Updates per month since 2007'''<br />
<br />
http://i.imgur.com/YWYd7.png<br />
<br />
'''Releases of Haskell libraries and tools per day, on hackage, since inception'''<br />
<br />
http://i.imgur.com/ws8rO.png<br />
<br />
'''Available Haskell packages over time'''<br />
<br />
http://i.imgur.com/aAbsc.png<br />
<br />
(Via the Haskell [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/gnuplot gnuplot] bindings).<br />
<br />
'''Statistics'''<br />
<br />
http://hackage.haskell.org/cgi-bin/hackage-scripts/stats<br />
<br />
Graphics generated by the [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hsparklines hsparklines] library, by way of [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hackage-sparks-0.1 hackage-sparks].</div>DonStewarthttps://wiki.haskell.org/index.php?title=Template:Main/News&diff=40552Template:Main/News2011-06-18T14:34:39Z<p>DonStewart: </p>
<hr />
<div><div class="subtitle">Recent Package Updates [http://haskell.org/haskellwiki/Hackage_statistics http://i.imgur.com/mHvNV.png] [http://hackage.haskell.org/packages/archive/recent.rss http://haskell.org/wikiupload/7/7c/Rss16.png]</div><br />
<br />
<div style="font-size:80%"><br />
;[http://hackage.haskell.org/package/LibClang-0.0.8 LibClang 0.0.8]<br />
:Haskell bindings for libclang (a C++ parsing library)<br />
;[http://hackage.haskell.org/package/acid-state-0.3.1 acid-state 0.3.1]<br />
:Add ACID guarantees to any serializable Haskell data structure.<br />
;[http://hackage.haskell.org/package/alsa-seq-0.5.1 alsa-seq 0.5.1]<br />
:Binding to the ALSA Library API (MIDI sequencer).<br />
;[http://hackage.haskell.org/package/enumset-0.0.2 enumset 0.0.2]<br />
:Sets of enumeration values represented by machine words<br />
;[http://hackage.haskell.org/package/utility-ht-0.0.7 utility-ht 0.0.7]<br />
:Various small helper functions for Lists, Maybes, Tuples, Functions<br />
;[http://hackage.haskell.org/package/text-json-qq-0.4.0 text-json-qq 0.4.0]<br />
:Json Quasiquatation for Haskell.<br />
;[http://hackage.haskell.org/package/aeson-qq-0.4.0 aeson-qq 0.4.0]<br />
:Json Quasiquatation for Haskell.<br />
;[http://hackage.haskell.org/package/json-qq-0.4.0 json-qq 0.4.0]<br />
:Json Quasiquatation library for Haskell.<br />
;[http://hackage.haskell.org/package/swish-0.3.0.3 swish 0.3.0.3]<br />
:A semantic web toolkit.<br />
;[http://hackage.haskell.org/package/wl-pprint-1.1 wl-pprint 1.1]<br />
:The Wadler/Leijen Pretty Printer<br />
;[http://hackage.haskell.org/package/test-framework-th-0.2.0 test-framework-th 0.2.0]<br />
:Automagically generate the HUnit- and Quickcheck-bulk-code using Template Haskell.<br />
;[http://hackage.haskell.org/package/language-haskell-extract-0.2.0 language-haskell-extract 0.2.0]<br />
:Module to automatically extract functions from the local code.<br />
;[http://hackage.haskell.org/package/functor-combo-0.1.0 functor-combo 0.1.0]<br />
:Functor combinators with tries & zippers<br />
;[http://hackage.haskell.org/package/xfconf-4.8.0.0 xfconf 4.8.0.0]<br />
:FFI bindings to xfconf<br />
;[http://hackage.haskell.org/package/pvd-1.1.1 pvd 1.1.1]<br />
:A photo viewer daemon application with remote controlling abilities.<br />
;[http://hackage.haskell.org/package/buildbox-tools-1.5.0.1 buildbox-tools 1.5.0.1]<br />
:Tools for working with buildbox benchmark result files.<br />
;[http://hackage.haskell.org/package/buildbox-1.5.0.1 buildbox 1.5.0.1]<br />
:Rehackable components for writing buildbots and test harnesses.<br />
[http://hackage.haskell.org/packages/archive/pkg-list.html More...]<br />
<br />
<br />
<br />
</div></div>DonStewarthttps://wiki.haskell.org/index.php?title=Hackage_statistics&diff=40551Hackage statistics2011-06-18T14:33:45Z<p>DonStewart: </p>
<hr />
<div>A [http://www.jasani.org/2008/02/27/initial-release-of-hsparklines-010 sparkline] depicting statistics about the Haskell library repository, http://hackage.haskell.org. <br />
<br />
'''Updates this month by day'''<br />
<br />
http://i.imgur.com/mHvNV.png<br />
<br />
'''Updates per month since 2007'''<br />
<br />
http://i.imgur.com/YWYd7.png<br />
<br />
'''Releases of Haskell libraries and tools per day, on hackage, since inception'''<br />
<br />
http://i.imgur.com/ws8rO.png<br />
<br />
'''Available Haskell packages over time'''<br />
<br />
http://i.imgur.com/Be4vW.png<br />
<br />
(Via the Haskell [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/gnuplot gnuplot] bindings).<br />
<br />
'''Statistics'''<br />
<br />
http://hackage.haskell.org/cgi-bin/hackage-scripts/stats<br />
<br />
Graphics generated by the [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hsparklines hsparklines] library, by way of [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hackage-sparks-0.1 hackage-sparks].</div>DonStewarthttps://wiki.haskell.org/index.php?title=Hackage_statistics&diff=40550Hackage statistics2011-06-18T14:02:39Z<p>DonStewart: </p>
<hr />
<div>A [http://www.jasani.org/2008/02/27/initial-release-of-hsparklines-010 sparkline] depicting statistics about the Haskell library repository, http://hackage.haskell.org. <br />
<br />
'''Updates this month by day'''<br />
<br />
* [http://haskell.org/haskellwiki/Hackage_statistics http://galois.com/~dons/images/hackage-daily.png]<br />
<br />
'''Updates per month since 2007'''<br />
* [http://haskell.org/haskellwiki/Hackage_statistics http://galois.com/~dons/images/hackage-monthly.png]<br />
<br />
'''Releases of Haskell libraries and tools per day, on hackage, since inception'''<br />
<br />
http://i.imgur.com/ws8rO.png<br />
<br />
'''Available Haskell packages over time'''<br />
<br />
http://i.imgur.com/Be4vW.png<br />
<br />
(Via the Haskell [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/gnuplot gnuplot] bindings).<br />
<br />
'''Statistics'''<br />
<br />
http://hackage.haskell.org/cgi-bin/hackage-scripts/stats<br />
<br />
Graphics generated by the [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hsparklines hsparklines] library, by way of [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hackage-sparks-0.1 hackage-sparks].</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40394Numeric Haskell: A Repa Tutorial2011-06-05T15:10:50Z<p>DonStewart: /* Mapping, with indices */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
''Authors: [http://donsbot.wordpress.com Don Stewart].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformly<br />
over arrays with different shape.<br />
<br />
=== Building shapes ===<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
=== Working with shapes ===<br />
<br />
That one key operation, '''extent''', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
-- build an array<br />
Repa> let x :: Array DIM3 Int; x = fromList (Z :. (3::Int) :. (3::Int) :. (3::Int)) [1..27]<br />
Repa> :t x<br />
x :: Array DIM3 Int<br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Generating random arrays ===<br />
<br />
The [http://hackage.haskell.org/package/repa-algorithms repa-algorithms] package lets us generate new arrays with random data:<br />
<br />
<haskell><br />
-- 3d array of Ints, bounded between 0 and 255.<br />
Repa> :m +Data.Array.Repa.Algorithms.Randomish<br />
Repa Randomish> randomishIntArray (Z :. (3::Int) :. (3::Int) :. (3::Int)) 0 255 1<br />
[217,42,130,200,216,254<br />
,67,77,152,85,140,226,179<br />
,71,23,17,152,84,47,17,45<br />
,5,88,245,107,214,136]<br />
</haskell><br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
=== Copying arrays from pointers ===<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An alternative is to use indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Mapping, with indices ===<br />
<br />
A very powerful operator is <code>traverse</code>, a parallel array traversal which also supplies the current index:<br />
<br />
<haskell><br />
traverse :: (Shape sh, Shape sh', Elt a) <br />
=> Array sh a <br />
-- Source array<br />
-> (sh -> sh') <br />
-- Function to produce the extent of the result.<br />
-> ((sh -> a) -> sh' -> b) <br />
-- Function to produce elements of the result.<br />
-- It is passed a lookup function to<br />
-- get elements of the source.<br />
-> Array sh' b<br />
</haskell><br />
<br />
This is quite a compicated type, because it is very general. Let's take it apart. The first argument is the source array, which is obvious. The second argument is a function that transforms the shape of the input array to yield the output array. So if the arrays are the same size, this function is <code>id</code>. It might grow or resize the shape in other ways.<br />
<br />
Finally, the 3rd argument is where the magic is. Given an index, return a new element, and you also get a lookup function which when applied yields the current element.<br />
<br />
So we see this generalizes map to support indexes, and optional inspection of the current element. Let's try some examples:<br />
<br />
<haskell><br />
$ ghci <br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
*> :m + Data.Array.Repa<br />
*> :m + Data.Array.Repa.Algorithms.Randomish<br />
*> let a :: Array DIM3 Int;<br />
a = fromList (Z :. (3::Int) :. (3::Int) :. (3::Int)) [1..27]<br />
*> a<br />
[1,2,3,4,5,6,7,8,9<br />
,10,11,12,13,14,15,16,17,18<br />
,19,20,21,22,23,24,25,26,27]<br />
<br />
-- Keeping the shape the same, and just overwriting elements<br />
-- Use `traverse` to set all elements to their `x` axis:<br />
*> traverse a id (\_ (Z :. i :. j :. k) -> i) <br />
[0,0,0,0,0,0,0,0,0<br />
,1,1,1,1,1,1,1,1,1<br />
,2,2,2,2,2,2,2,2,2]<br />
<br />
-- Shuffle elements around, based on their index.<br />
-- Rotate elements by swapping elements from rotated locations:<br />
> traverse a id (\f (Z :. i :. j :. k) -> f (Z :. j :. k :. i)) <br />
[1,4,7,10,13,16,19,22,25<br />
,2,5,8,11,14,17,20,23,26<br />
,3,6,9,12,15,18,21,24,27]<br />
</haskell><br />
<br />
The documentation on [http://hackage.haskell.org/packages/archive/repa/2.0.2.1/doc/html/Data-Array-Repa.html#g:7 traverse] provides further information.<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Example: Rotating an image with backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
[http://hackage.haskell.org/package/repa-algorithms repa-algorithms] package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to average RGB pixels into a common grey value, where the average is weighted for human perception of green.<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on perceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
== Optimising Repa programs ==<br />
<br />
=== Fusion, and why you need it ===<br />
Repa depends critically on array fusion to achieve fast code. Fusion is a fancy name for the combination of inlining and code transformations performed by GHC when it compiles your program. The fusion process merges the array filling loops defined in the Repa library, with the "worker" functions that you write in your own module. If the fusion process fails, then the resulting program will be much slower than it needs to be, often 10x slower an equivalent program using plain Haskell lists. On the other hand, provided fusion works, the resulting code will run as fast as an equivalent cleanly written C program. Making fusion work is not hard once you understand what's going on.<br />
<br />
=== The <code>force</code> function has the loops ===<br />
<br />
Suppose we have the following binding: <br />
<br />
arr' = R.force $ R.map (\x -> x + 1) arr<br />
<br />
The right of this binding will compile down to code that first allocates the result array <code>arr'</code>, then iterates over the source array <code>arr</code>reading each element in turn, adding one to it, then writing to the corresponding element in the result.<br />
<br />
Importantly, the code that does the allocation, iteration and update is defined as part of the <code>force</code> function. This forcing code has been written to break up the result into several chunks, and evaluate each chunk with a different thread. This is what makes your code run in parallel. If you do ''not'' use <code>force</code> then your code will be slow and ''not'' run in parallel.<br />
<br />
=== Delayed and Manifest arrays ===<br />
In the example from the previous section, think of the <code>R.map (\x -> x + 1) arr</code> expression as a ''specification'' for a new array. In the library, this specification is referred to as a ''delayed'' array. A delayed array is represented as a function that takes an array index, and computes the value of the element at that index.<br />
<br />
Applying <code>force</code> to a delayed array causes all elements to be computed in parallel. The result of a <code>force</code> is referred to as a ''manifest'' array. A manifest array is a "real" array represented as a flat chunk of memory containing array elements.<br />
<br />
All Repa array operators will accept both delayed and manifest arrays. However, if you index into a delayed array without forcing it first, then each indexing operation costs a function call. It also ''recomputes'' the value of the array element at that index.<br />
<br />
=== Shells and Springs ===<br />
Here is another way to think about Repa's approach to array fusion. Suppose we write the following binding:<br />
<br />
arr' = R.force $ R.map (\x -> x * 2) $ R.map (\x -> x + 1) arr<br />
<br />
Remember from the previous section, that the result of each of the applications of <code>R.map</code> is a delayed array. A delayed array is not a "real", manifest array, it's just a shell that contains a function to compute each element. In this example, the two worker functions correspond to the lambda expressions applied to <code>R.map</code>.<br />
<br />
When GHC compiles this example, the two worker functions are fused into a fresh unfolding of the parallel loop defined in the code for <code>R.force</code>. Imagine holding <code>R.force</code> in your left hand, and squashing the calls to <code>R.map</code> into it, like a spring. Doing this breaks all the shells, and you end up with the worker functions fused into an unfolding of <code>R.force</code>.<br />
<br />
=== INLINE worker functions ===<br />
Consider the following example:<br />
<br />
f x = x + 1<br />
arr' = R.force $ R.zipWith (*) (R.map f arr1) (R.map f arr2)<br />
<br />
During compilation, we need GHC to fuse our worker functions into a fresh unfolding of <code>R.force</code>. In this example, fusion includes inlining the definition of <code>f</code>. If <code>f</code> is ''not'' inlined, then the performance of the compiled code will be atrocious. It will perform a function call for each application of <code>f</code>, where it really only needs a single machine instruction to increment the <code>x</code> value.<br />
<br />
Now, in general, GHC tries to avoid producing binaries that are "too big". Part of this is a heuristic that controls exactly what functions are inlined. The heuristic says that a function may be inlined only if it is used once, or if its definition is less than some particular size. If neither of these apply, then the function won't be inlined, killing performance. <br />
<br />
For Repa programs, as fusion and inlining has such a dramatic effect on performance, we should ''absolutely not'' rely on heuristics to control whether or not this inlining takes place. If we rely on a heuristic, then even if our program runs fast today, if this heuristic is ever altered then some functions that used to be inlined may no longer be.<br />
<br />
The moral of the story is to attach INLINE pragmas to ''all'' of your client functions that compute array values. This ensures that these critical functions will be inlined now, and forever.<br />
<br />
{-# INLINE f #-}<br />
f x = x + 1<br />
<br />
arr' = R.force $ R.zipWith (*) (R.map f arr1) (R.map f arr2)<br />
<br />
== Advanced techniques ==<br />
<br />
=== Repa's parallel programming model ===<br />
<br />
''Discussion about the gang threads and hooks to help''<br />
<br />
=== Programming with stencils ===<br />
<br />
''Discuss the stencil types model''<br />
<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40392Numeric Haskell: A Repa Tutorial2011-06-05T14:41:56Z<p>DonStewart: /* Mapping, with indices */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
''Authors: [http://donsbot.wordpress.com Don Stewart].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformly<br />
over arrays with different shape.<br />
<br />
=== Building shapes ===<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
=== Working with shapes ===<br />
<br />
That one key operation, '''extent''', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
-- build an array<br />
Repa> let x :: Array DIM3 Int; x = fromList (Z :. (3::Int) :. (3::Int) :. (3::Int)) [1..27]<br />
Repa> :t x<br />
x :: Array DIM3 Int<br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Generating random arrays ===<br />
<br />
The [http://hackage.haskell.org/package/repa-algorithms repa-algorithms] package lets us generate new arrays with random data:<br />
<br />
<haskell><br />
-- 3d array of Ints, bounded between 0 and 255.<br />
Repa> :m +Data.Array.Repa.Algorithms.Randomish<br />
Repa Randomish> randomishIntArray (Z :. (3::Int) :. (3::Int) :. (3::Int)) 0 255 1<br />
[217,42,130,200,216,254<br />
,67,77,152,85,140,226,179<br />
,71,23,17,152,84,47,17,45<br />
,5,88,245,107,214,136]<br />
</haskell><br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
=== Copying arrays from pointers ===<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An alternative is to use indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Mapping, with indices ===<br />
<br />
A very powerful operator is <code>traverse</code>, a parallel array traversal which also supplies the current index:<br />
<br />
<haskell><br />
traverse :: (Shape sh, Shape sh', Elt a) <br />
=> Array sh a <br />
-- Source array<br />
-> (sh -> sh') <br />
-- Function to produce the extent of the result.<br />
-> ((sh -> a) -> sh' -> b) <br />
-- Function to produce elements of the result.<br />
-- It is passed a lookup function to<br />
-- get elements of the source.<br />
-> Array sh' b<br />
</haskell><br />
<br />
This is quite a compicated type, because it is very general. Let's take it apart. The first argument is the source array, which is obvious. The second argument is a function that transforms the shape of the input array to yield the output array. So if the arrays are the same size, this function is <code>id</code>. It might grow or resize the shape in other ways.<br />
<br />
Finally, the 3rd argument is where the magic is. Given an index, return a new element, and you also get a lookup function which when applied yields the current element.<br />
<br />
So we see this generalizes map to support indexes, and optional inspection of the current element. Let's try some examples:<br />
<br />
<haskell><br />
$ ghci <br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
*> :m + Data.Array.Repa<br />
*> :m + Data.Array.Repa.Algorithms.Randomish<br />
*> let a :: Array DIM3 Int;<br />
a = fromList (Z :. (3::Int) :. (3::Int) :. (3::Int)) [1..27]<br />
*> a<br />
[1,2,3,4,5,6,7,8,9<br />
,10,11,12,13,14,15,16,17,18<br />
,19,20,21,22,23,24,25,26,27]<br />
<br />
-- Keeping the shape the same, and just overwriting elements<br />
-- Use `traverse` to set all elements to their `x` axis:<br />
*> traverse a id (\_ (Z :. i :. j :. k) -> i) <br />
[0,0,0,0,0,0,0,0,0<br />
,1,1,1,1,1,1,1,1,1<br />
,2,2,2,2,2,2,2,2,2]<br />
<br />
-- Shuffle elements around, based on their index.<br />
-- Rotate elements by swapping elements from rotated locations:<br />
> traverse a id (\f (Z :. i :. j :. k) -> f (Z :. j :. k :. i)) <br />
[1,4,7,10,13,16,19,22,25<br />
,2,5,8,11,14,17,20,23,26<br />
,3,6,9,12,15,18,21,24,27]<br />
<br />
The documentation on [http://hackage.haskell.org/packages/archive/repa/2.0.2.1/doc/html/Data-Array-Repa.html#g:7 traverse] provides further information.<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Example: Rotating an image with backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
[http://hackage.haskell.org/package/repa-algorithms repa-algorithms] package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to average RGB pixels into a common grey value, where the average is weighted for human perception of green.<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on perceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
== Optimising Repa programs ==<br />
<br />
=== Fusion, and why you need it ===<br />
Repa depends critically on array fusion to achieve fast code. Fusion is a fancy name for the combination of inlining and code transformations performed by GHC when it compiles your program. The fusion process merges the array filling loops defined in the Repa library, with the "worker" functions that you write in your own module. If the fusion process fails, then the resulting program will be much slower than it needs to be, often 10x slower an equivalent program using plain Haskell lists. On the other hand, provided fusion works, the resulting code will run as fast as an equivalent cleanly written C program. Making fusion work is not hard once you understand what's going on.<br />
<br />
=== The <code>force</code> function has the loops ===<br />
<br />
Suppose we have the following binding: <br />
<br />
arr' = R.force $ R.map (\x -> x + 1) arr<br />
<br />
The right of this binding will compile down to code that first allocates the result array <code>arr'</code>, then iterates over the source array <code>arr</code>reading each element in turn, adding one to it, then writing to the corresponding element in the result.<br />
<br />
Importantly, the code that does the allocation, iteration and update is defined as part of the <code>force</code> function. This forcing code has been written to break up the result into several chunks, and evaluate each chunk with a different thread. This is what makes your code run in parallel. If you do ''not'' use <code>force</code> then your code will be slow and ''not'' run in parallel.<br />
<br />
=== Delayed and Manifest arrays ===<br />
In the example from the previous section, think of the <code>R.map (\x -> x + 1) arr</code> expression as a ''specification'' for a new array. In the library, this specification is referred to as a ''delayed'' array. A delayed array is represented as a function that takes an array index, and computes the value of the element at that index.<br />
<br />
Applying <code>force</code> to a delayed array causes all elements to be computed in parallel. The result of a <code>force</code> is referred to as a ''manifest'' array. A manifest array is a "real" array represented as a flat chunk of memory containing array elements.<br />
<br />
All Repa array operators will accept both delayed and manifest arrays. However, if you index into a delayed array without forcing it first, then each indexing operation costs a function call. It also ''recomputes'' the value of the array element at that index.<br />
<br />
=== Shells and Springs ===<br />
Here is another way to think about Repa's approach to array fusion. Suppose we write the following binding:<br />
<br />
arr' = R.force $ R.map (\x -> x * 2) $ R.map (\x -> x + 1) arr<br />
<br />
Remember from the previous section, that the result of each of the applications of <code>R.map</code> is a delayed array. A delayed array is not a "real", manifest array, it's just a shell that contains a function to compute each element. In this example, the two worker functions correspond to the lambda expressions applied to <code>R.map</code>.<br />
<br />
When GHC compiles this example, the two worker functions are fused into a fresh unfolding of the parallel loop defined in the code for <code>R.force</code>. Imagine holding <code>R.force</code> in your left hand, and squashing the calls to <code>R.map</code> into it, like a spring. Doing this breaks all the shells, and you end up with the worker functions fused into an unfolding of <code>R.force</code>.<br />
<br />
=== INLINE worker functions ===<br />
Consider the following example:<br />
<br />
f x = x + 1<br />
arr' = R.force $ R.zipWith (*) (R.map f arr1) (R.map f arr2)<br />
<br />
During compilation, we need GHC to fuse our worker functions into a fresh unfolding of <code>R.force</code>. In this example, fusion includes inlining the definition of <code>f</code>. If <code>f</code> is ''not'' inlined, then the performance of the compiled code will be atrocious. It will perform a function call for each application of <code>f</code>, where it really only needs a single machine instruction to increment the <code>x</code> value.<br />
<br />
Now, in general, GHC tries to avoid producing binaries that are "too big". Part of this is a heuristic that controls exactly what functions are inlined. The heuristic says that a function may be inlined only if it is used once, or if its definition is less than some particular size. If neither of these apply, then the function won't be inlined, killing performance. <br />
<br />
For Repa programs, as fusion and inlining has such a dramatic effect on performance, we should ''absolutely not'' rely on heuristics to control whether or not this inlining takes place. If we rely on a heuristic, then even if our program runs fast today, if this heuristic is ever altered then some functions that used to be inlined may no longer be.<br />
<br />
The moral of the story is to attach INLINE pragmas to ''all'' of your client functions that compute array values. This ensures that these critical functions will be inlined now, and forever.<br />
<br />
{-# INLINE f #-}<br />
f x = x + 1<br />
<br />
arr' = R.force $ R.zipWith (*) (R.map f arr1) (R.map f arr2)<br />
<br />
== Advanced techniques ==<br />
<br />
=== Repa's parallel programming model ===<br />
<br />
''Discussion about the gang threads and hooks to help''<br />
<br />
=== Programming with stencils ===<br />
<br />
''Discuss the stencil types model''<br />
<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40391Numeric Haskell: A Repa Tutorial2011-06-05T14:31:23Z<p>DonStewart: /* Numeric operations: negation, addition, subtraction, multiplication */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
''Authors: [http://donsbot.wordpress.com Don Stewart].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformly<br />
over arrays with different shape.<br />
<br />
=== Building shapes ===<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
=== Working with shapes ===<br />
<br />
That one key operation, '''extent''', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
-- build an array<br />
Repa> let x :: Array DIM3 Int; x = fromList (Z :. (3::Int) :. (3::Int) :. (3::Int)) [1..27]<br />
Repa> :t x<br />
x :: Array DIM3 Int<br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Generating random arrays ===<br />
<br />
The [http://hackage.haskell.org/package/repa-algorithms repa-algorithms] package lets us generate new arrays with random data:<br />
<br />
<haskell><br />
-- 3d array of Ints, bounded between 0 and 255.<br />
Repa> :m +Data.Array.Repa.Algorithms.Randomish<br />
Repa Randomish> randomishIntArray (Z :. (3::Int) :. (3::Int) :. (3::Int)) 0 255 1<br />
[217,42,130,200,216,254<br />
,67,77,152,85,140,226,179<br />
,71,23,17,152,84,47,17,45<br />
,5,88,245,107,214,136]<br />
</haskell><br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
=== Copying arrays from pointers ===<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An alternative is to use indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Mapping, with indices ===<br />
<br />
A very powerful operator is <code>traverse</code>, a parallel array traversal which also supplies the current index:<br />
<br />
<haskell><br />
traverse :: (Shape sh, Shape sh', Elt a) <br />
=> Array sh a -- Source array<br />
-> (sh -> sh') -- Function to produce the extent of the result.<br />
-> ((sh -> a) -> sh' -> b) -- Function to produce elements of the result.<br />
-- It is passed a lookup function to get elements of the source.<br />
-> Array sh' b<br />
</haskell><br />
<br />
This is quite a compicated type, because it is very general. Let's take it apart. The first argument is the source array, which is obvious. The second argument is a function that transforms the shape of the input array to yield the output array. So if the arrays are the same size, this function is <code>id</code>. It might grow or resize the shape in other ways.<br />
<br />
Finally, the 3rd argument is where the magic is. Given an index, return a new element, and you also get a lookup function which when applied yields the current element.<br />
<br />
So we see this generalizes map to support indexes, and optional inspection of the current element. Let's try some examples:<br />
<br />
<br />
<br />
The documentation on [http://hackage.haskell.org/packages/archive/repa/2.0.2.1/doc/html/Data-Array-Repa.html#g:7 traverse] provides further information.<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Example: Rotating an image with backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
[http://hackage.haskell.org/package/repa-algorithms repa-algorithms] package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to average RGB pixels into a common grey value, where the average is weighted for human perception of green.<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on perceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
== Optimising Repa programs ==<br />
<br />
=== Fusion, and why you need it ===<br />
Repa depends critically on array fusion to achieve fast code. Fusion is a fancy name for the combination of inlining and code transformations performed by GHC when it compiles your program. The fusion process merges the array filling loops defined in the Repa library, with the "worker" functions that you write in your own module. If the fusion process fails, then the resulting program will be much slower than it needs to be, often 10x slower an equivalent program using plain Haskell lists. On the other hand, provided fusion works, the resulting code will run as fast as an equivalent cleanly written C program. Making fusion work is not hard once you understand what's going on.<br />
<br />
=== The <code>force</code> function has the loops ===<br />
<br />
Suppose we have the following binding: <br />
<br />
arr' = R.force $ R.map (\x -> x + 1) arr<br />
<br />
The right of this binding will compile down to code that first allocates the result array <code>arr'</code>, then iterates over the source array <code>arr</code>reading each element in turn, adding one to it, then writing to the corresponding element in the result.<br />
<br />
Importantly, the code that does the allocation, iteration and update is defined as part of the <code>force</code> function. This forcing code has been written to break up the result into several chunks, and evaluate each chunk with a different thread. This is what makes your code run in parallel. If you do ''not'' use <code>force</code> then your code will be slow and ''not'' run in parallel.<br />
<br />
=== Delayed and Manifest arrays ===<br />
In the example from the previous section, think of the <code>R.map (\x -> x + 1) arr</code> expression as a ''specification'' for a new array. In the library, this specification is referred to as a ''delayed'' array. A delayed array is represented as a function that takes an array index, and computes the value of the element at that index.<br />
<br />
Applying <code>force</code> to a delayed array causes all elements to be computed in parallel. The result of a <code>force</code> is referred to as a ''manifest'' array. A manifest array is a "real" array represented as a flat chunk of memory containing array elements.<br />
<br />
All Repa array operators will accept both delayed and manifest arrays. However, if you index into a delayed array without forcing it first, then each indexing operation costs a function call. It also ''recomputes'' the value of the array element at that index.<br />
<br />
=== Shells and Springs ===<br />
Here is another way to think about Repa's approach to array fusion. Suppose we write the following binding:<br />
<br />
arr' = R.force $ R.map (\x -> x * 2) $ R.map (\x -> x + 1) arr<br />
<br />
Remember from the previous section, that the result of each of the applications of <code>R.map</code> is a delayed array. A delayed array is not a "real", manifest array, it's just a shell that contains a function to compute each element. In this example, the two worker functions correspond to the lambda expressions applied to <code>R.map</code>.<br />
<br />
When GHC compiles this example, the two worker functions are fused into a fresh unfolding of the parallel loop defined in the code for <code>R.force</code>. Imagine holding <code>R.force</code> in your left hand, and squashing the calls to <code>R.map</code> into it, like a spring. Doing this breaks all the shells, and you end up with the worker functions fused into an unfolding of <code>R.force</code>.<br />
<br />
=== INLINE worker functions ===<br />
Consider the following example:<br />
<br />
f x = x + 1<br />
arr' = R.force $ R.zipWith (*) (R.map f arr1) (R.map f arr2)<br />
<br />
During compilation, we need GHC to fuse our worker functions into a fresh unfolding of <code>R.force</code>. In this example, fusion includes inlining the definition of <code>f</code>. If <code>f</code> is ''not'' inlined, then the performance of the compiled code will be atrocious. It will perform a function call for each application of <code>f</code>, where it really only needs a single machine instruction to increment the <code>x</code> value.<br />
<br />
Now, in general, GHC tries to avoid producing binaries that are "too big". Part of this is a heuristic that controls exactly what functions are inlined. The heuristic says that a function may be inlined only if it is used once, or if its definition is less than some particular size. If neither of these apply, then the function won't be inlined, killing performance. <br />
<br />
For Repa programs, as fusion and inlining has such a dramatic effect on performance, we should ''absolutely not'' rely on heuristics to control whether or not this inlining takes place. If we rely on a heuristic, then even if our program runs fast today, if this heuristic is ever altered then some functions that used to be inlined may no longer be.<br />
<br />
The moral of the story is to attach INLINE pragmas to ''all'' of your client functions that compute array values. This ensures that these critical functions will be inlined now, and forever.<br />
<br />
{-# INLINE f #-}<br />
f x = x + 1<br />
<br />
arr' = R.force $ R.zipWith (*) (R.map f arr1) (R.map f arr2)<br />
<br />
== Advanced techniques ==<br />
<br />
=== Repa's parallel programming model ===<br />
<br />
''Discussion about the gang threads and hooks to help''<br />
<br />
=== Programming with stencils ===<br />
<br />
''Discuss the stencil types model''<br />
<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40082Numeric Haskell: A Repa Tutorial2011-05-24T18:46:57Z<p>DonStewart: /* Delayed and manifest arrays */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
''Authors: [http://donsbot.wordpress.com Don Stewart].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformly<br />
over arrays with different shape.<br />
<br />
=== Building shapes ===<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
=== Working with shapes ===<br />
<br />
That one key operation, '''extent''', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
-- build an array<br />
Repa> let x :: Array DIM3 Int; x = fromList (Z :. (3::Int) :. (3::Int) :. (3::Int)) [1..27]<br />
Repa> :t x<br />
x :: Array DIM3 Int<br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Generating random arrays ===<br />
<br />
The [http://hackage.haskell.org/package/repa-algorithms repa-algorithms] package lets us generate new arrays with random data:<br />
<br />
<haskell><br />
-- 3d array of Ints, bounded between 0 and 255.<br />
Repa> :m +Data.Array.Repa.Algorithms.Randomish<br />
Repa Randomish> randomishIntArray (Z :. (3::Int) :. (3::Int) :. (3::Int)) 0 255 1<br />
[217,42,130,200,216,254<br />
,67,77,152,85,140,226,179<br />
,71,23,17,152,84,47,17,45<br />
,5,88,245,107,214,136]<br />
</haskell><br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
=== Copying arrays from pointers ===<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An alternative is to use indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Example: Rotating an image with backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
[http://hackage.haskell.org/package/repa-algorithms repa-algorithms] package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to average RGB pixels into a common grey value, where the average is weighted for human perception of green.<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on perceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
== Advanced techniques ==<br />
<br />
=== Repa's parallel programming model ===<br />
<br />
''Discussion about the gang threads and hooks to help''<br />
<br />
=== Programming with stencils ===<br />
<br />
''Discuss the stencil types model''<br />
<br />
=== Delayed and manifest arrays ===<br />
<br />
''When to write `force`''<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40081Numeric Haskell: A Repa Tutorial2011-05-24T18:41:49Z<p>DonStewart: /* Example: parallel image desaturation */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
''Authors: [http://donsbot.wordpress.com Don Stewart].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformly<br />
over arrays with different shape.<br />
<br />
=== Building shapes ===<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
=== Working with shapes ===<br />
<br />
That one key operation, '''extent''', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
-- build an array<br />
Repa> let x :: Array DIM3 Int; x = fromList (Z :. (3::Int) :. (3::Int) :. (3::Int)) [1..27]<br />
Repa> :t x<br />
x :: Array DIM3 Int<br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Generating random arrays ===<br />
<br />
The [http://hackage.haskell.org/package/repa-algorithms repa-algorithms] package lets us generate new arrays with random data:<br />
<br />
<haskell><br />
-- 3d array of Ints, bounded between 0 and 255.<br />
Repa> :m +Data.Array.Repa.Algorithms.Randomish<br />
Repa Randomish> randomishIntArray (Z :. (3::Int) :. (3::Int) :. (3::Int)) 0 255 1<br />
[217,42,130,200,216,254<br />
,67,77,152,85,140,226,179<br />
,71,23,17,152,84,47,17,45<br />
,5,88,245,107,214,136]<br />
</haskell><br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
=== Copying arrays from pointers ===<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An alternative is to use indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Example: Rotating an image with backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
[http://hackage.haskell.org/package/repa-algorithms repa-algorithms] package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to average RGB pixels into a common grey value, where the average is weighted for human perception of green.<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on perceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
== Advanced techniques ==<br />
<br />
=== Repa's parallel programming model ===<br />
<br />
''Discussion about the gang threads and hooks to help''<br />
<br />
=== Programming with stencils ===<br />
<br />
''Discuss the stencil types model''<br />
<br />
=== Delayed and manifest arrays ===<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=IRC_channel&diff=40065IRC channel2011-05-19T20:37:01Z<p>DonStewart: </p>
<hr />
<div>Internet Relay Chat is a worldwide text chat service with many thousands<br />
of users among various irc networks.<br />
<br />
The Freenode IRC network hosts the very large #haskell channel, and we've had<br />
up to <br />
804<br />
concurrent users<br />
(average is 714<br />
), making the channel<br />
[http://searchirc.com/search.php?SCHANS=1&SSORT=SIZE&N=freenode one of the 10 largest]<br />
of the 12,000+ channels on freenode and the second largest programming language channel after Python. One famous<br />
resident is [[Lambdabot]], another is [http://hpaste.org hpaste] (see<br />
the [[#Bots|Bots]] section below).<br />
<br />
The IRC channel can be an excellent place to learn more about Haskell,<br />
and to just keep in the loop on new things in the Haskell world. Many<br />
new developments in the Haskell world first appear on the irc channel.<br />
<br />
Since 2009, the Haskell channel has grown large enough that we've split it in two parts:<br />
<br />
* #haskell, for all the usual things<br />
* #haskell-in-depth , for those seeking in depth, or more theoretical discussion<br />
<br />
As always, #haskell remains the primary place for new user questions.<br />
<br />
--------------------------<br />
<br />
[http://haskell.org/sitewiki/images/3/3c/Haskell-current.png [[Image:Haskell-current-small.png|thumb|The #haskell social graph, Jan 2008]]]<br />
<br />
[[Image:Irc-raw.png|thumb|Daily traffic in #haskell since 2004]]<br />
<br />
[[Image:Nick-activity.png|thumb|Growth of #haskell]]<br />
<br />
[[Image:Haskell-wordle-irc.png|thumb|Haskell noun map]]<br />
<br />
== Getting there ==<br />
<br />
If you point your irc client to [irc://chat.freenode.net/haskell chat.freenode.net] and then join the #haskell channel, you'll be there. Alternately, you can try http://java.freenode.net/ or http://webchat.freenode.net/ which connects inside the browser.<br />
<br />
Example, using [http://www.irssi.org/ irssi]:<br />
<br />
$ irssi -c chat.freenode.net -n myname -w mypassword<br />
/join #haskell<br />
<br />
Tip, if you're using Emacs to edit your Haskell sources then why not use it to chat about Haskell? Check out [http://www.emacswiki.org/cgi-bin/wiki/EmacsIRCClient ERC], The Emacs IRC client. Invoke it like this and follow the commands:<br />
<br />
M-x erc-select<br />
...<br />
/join #haskell<br />
<br />
[[Image:Irc--haskell-screenshot.png|frame|A screenshot of an irssi session in #haskell]]<br />
<br />
== Principles ==<br />
<br />
The #haskell channel is a very friendly, welcoming place to hang out,<br />
teach and learn. The goal of #haskell is to encourage learning and<br />
discussion of Haskell, functional programming, and programming in<br />
general. As part of this we welcome newbies, and encourage teaching of<br />
the language.<br />
<br />
Part of the #haskell success comes from the approach that the community<br />
is quite tight knit -- we know each other -- it's not just a homework<br />
channel. As a result, many collaborative projects have arisen between<br />
Haskell irc channel citizens.<br />
<br />
To maintain the friendly, open culture, the following is required:<br />
<br />
* Low to zero tolerance for ridiculing questions. Insulting new users is unacceptable.<br />
<br />
New Haskell users should feel entirely comfortable asking new questions.<br />
Helpful answers should be encouraged with <hask>name++</hask> karma<br />
points, in public, as a reward for providing a good answer.<br />
<br />
As the channel grows, we see a diverse range of people, with different<br />
programming backgrounds, trying to make their way with Haskell. A good<br />
rule of thumb, to avoid frustration is:<br />
<br />
* approach negative comments by asking for details (kind of like [http://en.wikipedia.org/wiki/Socratic_method Socratic questioning]), rather than challenging the competence of the writer (ad hominem).<br />
<br />
<br />
== History ==<br />
<br />
The #haskell channel appeared in the late 90s, and really got going<br />
in early 2001, with the help of Shae Erisson (aka shapr).<br />
<br />
A fairly extensive analysis of the traffic on #haskell over the years is<br />
[http://www.cse.unsw.edu.au/~dons/irc/ kept here]<br />
<br />
== Related channels ==<br />
<br />
In addition to the main Haskell channel there are also:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Channel<br />
! Purpose<br />
|-<br />
| #haskell.cz<br />
| Czech speakers (UTF-8)<br />
|- <br />
| #haskell.de<br />
| German speakers<br />
|-<br />
| #haskell.dut<br />
| Dutch speakers<br />
|-<br />
| #haskell.es<br />
| Spanish speakers<br />
|-<br />
| #haskell.fi<br />
| Finnish speakers<br />
|-<br />
| #haskell.fr <br />
| French speakers <br />
|-<br />
| #haskell.hr<br />
| Croatian speakers<br />
|-<br />
| #haskell.it <br />
| Italian speakers<br />
|-<br />
| #haskell.jp <br />
| Japanese speakers<br />
|-<br />
| #haskell.no <br />
| Norwegian speakers<br />
|-<br />
| #haskell.pt<br />
| Portuguese speakers<br />
|-<br />
| #haskell.ru <br />
| Russian speakers. Seems that most of them migrated to Jabber conference (haskell@conference.jabber.ru).<br />
|-<br />
| #haskell_ru <br />
| Russian speakers again, in UTF-8. For those, who prefer good ol' IRC channel with a lambdabot.<br />
|-<br />
| #haskell.se <br />
| Swedish speakers<br />
|-<br />
| #haskell-blah <br />
| Haskell people talking about anything except Haskell itself<br />
|-<br />
| #haskell-books <br />
| Authors organizing the collaborative writing of the [http://en.wikibooks.org/wiki/Haskell Haskell wikibook] and other books or tutorials.<br />
|-<br />
| #haskell-game<br />
| The hub for Haskell-based [[Game Development|game development]]<br />
|-<br />
| #haskell-in-depth<br />
| slower paced discussion of use, theory, implementation etc with no monad tutorials!<br />
|-<br />
| #haskell-iphone<br />
| Haskell-based [[iPhone]] development<br />
|-<br />
| #haskell-overflow<br />
| Overflow conversations<br />
|-<br />
| #haskell-web<br />
| Friendly, practical discussion of haskell web app/framework/server development<br />
|-<br />
| '''Platform-specific:'''<br />
|<br />
|-<br />
| #arch-haskell <br />
| [[Arch Linux]]/ specific Haskell conversations<br />
|-<br />
| #gentoo-haskell <br />
| [[Gentoo]]/Linux specific Haskell conversations<br />
|-<br />
| '''Projects using haskell:'''<br />
|<br />
|-<br />
| #darcs <br />
| [[Darcs]] revision control system<br />
|-<br />
| #hackage<br />
| Haskell's software distribution infrastructure<br />
|-<br />
| #happs<br />
| [http://happstack.com Happstack] web framework<br />
|-<br />
| #hledger<br />
| [http://hledger.org hledger] accounting tools and library<br />
|-<br />
| #leksah<br />
| [http://leksah.org Leksah] IDE for Haskell development<br />
|-<br />
| #perl6 <br />
| [http://www.pugscode.org Perl 6] development (plenty of Haskell chat there too)<br />
|-<br />
| #snapframework<br />
| [http://snapframework.com/ Snap] web framework<br />
|-<br />
| #xmonad<br />
| [http://xmonad.org Xmonad] tiling window manager<br />
|}<br />
<br />
== Logs ==<br />
<br />
'''Logs''' are kept at a few places, including<br />
<br />
* [http://tunes.org/~nef/logs/haskell/ tunes.org]<br />
* [http://ircbrowse.com/cdates.html?channel=haskell IRCBrowse]<br />
<br />
<br />
== Bots ==<br />
<br />
There are various bots on the channel. Their names and usage are described here.<br />
<br />
=== lambdabot ===<br />
<br />
[[Lambdabot]] is both the name of a software package and a bot on the channel. It provides many useful services for visitors to the IRC channel. It is available as a haskell package and can be integrated into ghci. Details on the software are found on a [[Lambdabot|separate wiki page]].<br />
<br />
Here is its interface for the IRC user:<br />
<br />
lambdabot's commands are prepended by a '@' sign.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Command<br />
! Usage<br />
|-<br />
| @help<br />
| display help to other commands, but help text is not available for all commands.<br />
|-<br />
| @type EXPR or ':t' EXPR<br />
| shows the type of an expression<br />
|-<br />
| @kind TYPECONSTRUCTOR<br />
| shows the kind of a type constructor<br />
|-<br />
| @run EXPR or '>' EXPR<br />
| evaluates EXPR<br />
|-<br />
| @pl FUNCTION<br />
| shows a [[pointfree]] version of FUNCTION<br />
|-<br />
| @pointful FUNCTION or '@unpl' FUNCTION<br />
| shows a 'pointful' version of FUNCTION<br />
|}<br />
<br />
=== preflex ===<br />
<br />
is the name of a lambdabot with more commands/plugins enabled. It is run by ?? To talk to preflex, write <tt>preflex: command ARGS</tt><br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Command<br />
! Usage<br />
|-<br />
| help COMMAND<br />
| displays help to other commands.<br />
|-<br />
| list<br />
| lists all plugins with their commands<br />
|-<br />
| NICK++ / NICK--<br />
| in/decrements the karma of NICK.<br />
|-<br />
| karma NICK<br />
| shows the karma of NICK<br />
|-<br />
| seen NICK<br />
| shows information about the last message of a user<br />
|-<br />
| tell / ask<br />
| sends NICK MSG a message when she becomes active.<br />
|-<br />
| xseen<br />
| ''see 'seen' ?? any difference ?''<br />
|-<br />
| quote NICK<br />
| prints a random quote of NICK<br />
|-<br />
| remember NAME QUOTE<br />
| associates NAME with quote. can be accessed by 'quote'<br />
|-<br />
| ...<br />
| ...<br />
|}<br />
<br />
=== hpaste ===<br />
The hpaste bot provides a notification interface to the [http://hpaste.org hpaste pastebin]. [[Hpaste.el|Emacs integration]] is available.<br />
<br />
''Usage?''<br />
<br />
''Not online often !? ''<br />
<br />
=== hackage ===<br />
The hackage bot provides real-time notifications of new package uploads to [http://hackage.haskell.org Hackage].<br />
<br />
== Locations ==<br />
<br />
To get an overview of where everybody on the channel might<br />
be, physically, please visit [[Haskell user locations]].<br />
<br />
<br />
[[Category:Community]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=IRC_channel&diff=40042IRC channel2011-05-18T20:47:44Z<p>DonStewart: </p>
<hr />
<div>Internet Relay Chat is a worldwide text chat service with many thousands<br />
of users among various irc networks.<br />
<br />
The Freenode IRC network hosts the very large #haskell channel, and we've had<br />
up to <br />
793<br />
concurrent users<br />
(average is 714<br />
), making the channel<br />
[http://searchirc.com/search.php?SCHANS=1&SSORT=SIZE&N=freenode one of the 10 largest]<br />
of the 12,000+ channels on freenode and the second largest programming language channel after Python. One famous<br />
resident is [[Lambdabot]], another is [http://hpaste.org hpaste] (see<br />
the [[#Bots|Bots]] section below).<br />
<br />
The IRC channel can be an excellent place to learn more about Haskell,<br />
and to just keep in the loop on new things in the Haskell world. Many<br />
new developments in the Haskell world first appear on the irc channel.<br />
<br />
Since 2009, the Haskell channel has grown large enough that we've split it in two parts:<br />
<br />
* #haskell, for all the usual things<br />
* #haskell-in-depth , for those seeking in depth, or more theoretical discussion<br />
<br />
As always, #haskell remains the primary place for new user questions.<br />
<br />
--------------------------<br />
<br />
[http://haskell.org/sitewiki/images/3/3c/Haskell-current.png [[Image:Haskell-current-small.png|thumb|The #haskell social graph, Jan 2008]]]<br />
<br />
[[Image:Irc-raw.png|thumb|Daily traffic in #haskell since 2004]]<br />
<br />
[[Image:Nick-activity.png|thumb|Growth of #haskell]]<br />
<br />
[[Image:Haskell-wordle-irc.png|thumb|Haskell noun map]]<br />
<br />
== Getting there ==<br />
<br />
If you point your irc client to [irc://chat.freenode.net/haskell chat.freenode.net] and then join the #haskell channel, you'll be there. Alternately, you can try http://java.freenode.net/ or http://webchat.freenode.net/ which connects inside the browser.<br />
<br />
Example, using [http://www.irssi.org/ irssi]:<br />
<br />
$ irssi -c chat.freenode.net -n myname -w mypassword<br />
/join #haskell<br />
<br />
Tip, if you're using Emacs to edit your Haskell sources then why not use it to chat about Haskell? Check out [http://www.emacswiki.org/cgi-bin/wiki/EmacsIRCClient ERC], The Emacs IRC client. Invoke it like this and follow the commands:<br />
<br />
M-x erc-select<br />
...<br />
/join #haskell<br />
<br />
[[Image:Irc--haskell-screenshot.png|frame|A screenshot of an irssi session in #haskell]]<br />
<br />
== Principles ==<br />
<br />
The #haskell channel is a very friendly, welcoming place to hang out,<br />
teach and learn. The goal of #haskell is to encourage learning and<br />
discussion of Haskell, functional programming, and programming in<br />
general. As part of this we welcome newbies, and encourage teaching of<br />
the language.<br />
<br />
Part of the #haskell success comes from the approach that the community<br />
is quite tight knit -- we know each other -- it's not just a homework<br />
channel. As a result, many collaborative projects have arisen between<br />
Haskell irc channel citizens.<br />
<br />
To maintain the friendly, open culture, the following is required:<br />
<br />
* Low to zero tolerance for ridiculing questions. Insulting new users is unacceptable.<br />
<br />
New Haskell users should feel entirely comfortable asking new questions.<br />
Helpful answers should be encouraged with <hask>name++</hask> karma<br />
points, in public, as a reward for providing a good answer.<br />
<br />
As the channel grows, we see a diverse range of people, with different<br />
programming backgrounds, trying to make their way with Haskell. A good<br />
rule of thumb, to avoid frustration is:<br />
<br />
* approach negative comments by asking for details (kind of like [http://en.wikipedia.org/wiki/Socratic_method Socratic questioning]), rather than challenging the competence of the writer (ad hominem).<br />
<br />
<br />
== History ==<br />
<br />
The #haskell channel appeared in the late 90s, and really got going<br />
in early 2001, with the help of Shae Erisson (aka shapr).<br />
<br />
A fairly extensive analysis of the traffic on #haskell over the years is<br />
[http://www.cse.unsw.edu.au/~dons/irc/ kept here]<br />
<br />
== Related channels ==<br />
<br />
In addition to the main Haskell channel there are also:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Channel<br />
! Purpose<br />
|-<br />
| #haskell.cz<br />
| Czech speakers (UTF-8)<br />
|- <br />
| #haskell.de<br />
| German speakers<br />
|-<br />
| #haskell.dut<br />
| Dutch speakers<br />
|-<br />
| #haskell.es<br />
| Spanish speakers<br />
|-<br />
| #haskell.fi<br />
| Finnish speakers<br />
|-<br />
| #haskell.fr <br />
| French speakers <br />
|-<br />
| #haskell.hr<br />
| Croatian speakers<br />
|-<br />
| #haskell.it <br />
| Italian speakers<br />
|-<br />
| #haskell.jp <br />
| Japanese speakers<br />
|-<br />
| #haskell.no <br />
| Norwegian speakers<br />
|-<br />
| #haskell.pt<br />
| Portuguese speakers<br />
|-<br />
| #haskell.ru <br />
| Russian speakers. Seems that most of them migrated to Jabber conference (haskell@conference.jabber.ru).<br />
|-<br />
| #haskell_ru <br />
| Russian speakers again, in UTF-8. For those, who prefer good ol' IRC channel with a lambdabot.<br />
|-<br />
| #haskell.se <br />
| Swedish speakers<br />
|-<br />
| #haskell-blah <br />
| Haskell people talking about anything except Haskell itself<br />
|-<br />
| #haskell-books <br />
| Authors organizing the collaborative writing of the [http://en.wikibooks.org/wiki/Haskell Haskell wikibook] and other books or tutorials.<br />
|-<br />
| #haskell-game<br />
| The hub for Haskell-based [[Game Development|game development]]<br />
|-<br />
| #haskell-in-depth<br />
| slower paced discussion of use, theory, implementation etc with no monad tutorials!<br />
|-<br />
| #haskell-iphone<br />
| Haskell-based [[iPhone]] development<br />
|-<br />
| #haskell-overflow<br />
| Overflow conversations<br />
|-<br />
| #haskell-web<br />
| Friendly, practical discussion of haskell web app/framework/server development<br />
|-<br />
| '''Platform-specific:'''<br />
|<br />
|-<br />
| #arch-haskell <br />
| [[Arch Linux]]/ specific Haskell conversations<br />
|-<br />
| #gentoo-haskell <br />
| [[Gentoo]]/Linux specific Haskell conversations<br />
|-<br />
| '''Projects using haskell:'''<br />
|<br />
|-<br />
| #darcs <br />
| [[Darcs]] revision control system<br />
|-<br />
| #hackage<br />
| Haskell's software distribution infrastructure<br />
|-<br />
| #happs<br />
| [http://happstack.com Happstack] web framework<br />
|-<br />
| #hledger<br />
| [http://hledger.org hledger] accounting tools and library<br />
|-<br />
| #leksah<br />
| [http://leksah.org Leksah] IDE for Haskell development<br />
|-<br />
| #perl6 <br />
| [http://www.pugscode.org Perl 6] development (plenty of Haskell chat there too)<br />
|-<br />
| #snapframework<br />
| [http://snapframework.com/ Snap] web framework<br />
|-<br />
| #xmonad<br />
| [http://xmonad.org Xmonad] tiling window manager<br />
|}<br />
<br />
== Logs ==<br />
<br />
'''Logs''' are kept at a few places, including<br />
<br />
* [http://tunes.org/~nef/logs/haskell/ tunes.org]<br />
* [http://ircbrowse.com/cdates.html?channel=haskell IRCBrowse]<br />
<br />
<br />
== Bots ==<br />
<br />
There are various bots on the channel. Their names and usage are described here.<br />
<br />
=== lambdabot ===<br />
<br />
[[Lambdabot]] is both the name of a software package and a bot on the channel. It provides many useful services for visitors to the IRC channel. It is available as a haskell package and can be integrated into ghci. Details on the software are found on a [[Lambdabot|separate wiki page]].<br />
<br />
Here is its interface for the IRC user:<br />
<br />
lambdabot's commands are prepended by a '@' sign.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Command<br />
! Usage<br />
|-<br />
| @help<br />
| display help to other commands, but help text is not available for all commands.<br />
|-<br />
| @type EXPR or ':t' EXPR<br />
| shows the type of an expression<br />
|-<br />
| @kind TYPECONSTRUCTOR<br />
| shows the kind of a type constructor<br />
|-<br />
| @run EXPR or '>' EXPR<br />
| evaluates EXPR<br />
|-<br />
| @pl FUNCTION<br />
| shows a [[pointfree]] version of FUNCTION<br />
|-<br />
| @pointful FUNCTION or '@unpl' FUNCTION<br />
| shows a 'pointful' version of FUNCTION<br />
|}<br />
<br />
=== preflex ===<br />
<br />
is the name of a lambdabot with more commands/plugins enabled. It is run by ?? To talk to preflex, write <tt>preflex: command ARGS</tt><br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Command<br />
! Usage<br />
|-<br />
| help COMMAND<br />
| displays help to other commands.<br />
|-<br />
| list<br />
| lists all plugins with their commands<br />
|-<br />
| NICK++ / NICK--<br />
| in/decrements the karma of NICK.<br />
|-<br />
| karma NICK<br />
| shows the karma of NICK<br />
|-<br />
| seen NICK<br />
| shows information about the last message of a user<br />
|-<br />
| tell / ask<br />
| sends NICK MSG a message when she becomes active.<br />
|-<br />
| xseen<br />
| ''see 'seen' ?? any difference ?''<br />
|-<br />
| quote NICK<br />
| prints a random quote of NICK<br />
|-<br />
| remember NAME QUOTE<br />
| associates NAME with quote. can be accessed by 'quote'<br />
|-<br />
| ...<br />
| ...<br />
|}<br />
<br />
=== hpaste ===<br />
The hpaste bot provides a notification interface to the [http://hpaste.org hpaste pastebin]. [[Hpaste.el|Emacs integration]] is available.<br />
<br />
''Usage?''<br />
<br />
''Not online often !? ''<br />
<br />
=== hackage ===<br />
The hackage bot provides real-time notifications of new package uploads to [http://hackage.haskell.org Hackage].<br />
<br />
== Locations ==<br />
<br />
To get an overview of where everybody on the channel might<br />
be, physically, please visit [[Haskell user locations]].<br />
<br />
<br />
[[Category:Community]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40026Numeric Haskell: A Repa Tutorial2011-05-17T20:10:07Z<p>DonStewart: </p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
''Authors: [http://donsbot.wordpress.com Don Stewart].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
=== Building shapes ===<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
=== Working with shapes ===<br />
<br />
That one key operation, '''extent''', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
-- build an array<br />
Repa> let x :: Array DIM3 Int; x = fromList (Z :. (3::Int) :. (3::Int) :. (3::Int)) [1..27]<br />
Repa> :t x<br />
x :: Array DIM3 Int<br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Generating random arrays ===<br />
<br />
The [http://hackage.haskell.org/package/repa-algorithsm repa-algorithms] package lets us generate new arrays with random data:<br />
<br />
<haskell><br />
-- 3d array of Ints, bounded between 0 and 255.<br />
> randomishIntArray (Z :. (3::Int) :. (3::Int) :. (3::Int)) 0 255 1<br />
[217,42,130,200,216,254<br />
,67,77,152,85,140,226,179<br />
,71,23,17,152,84,47,17,45<br />
,5,88,245,107,214,136]<br />
</haskell><br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
=== Copying arrays from pointers ===<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Example: Rotating an image with backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40025Numeric Haskell: A Repa Tutorial2011-05-17T20:09:52Z<p>DonStewart: </p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
'''Authors: [http://donsbot.wordpress.com Don Stewart]'''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
=== Building shapes ===<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
=== Working with shapes ===<br />
<br />
That one key operation, '''extent''', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
-- build an array<br />
Repa> let x :: Array DIM3 Int; x = fromList (Z :. (3::Int) :. (3::Int) :. (3::Int)) [1..27]<br />
Repa> :t x<br />
x :: Array DIM3 Int<br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Generating random arrays ===<br />
<br />
The [http://hackage.haskell.org/package/repa-algorithsm repa-algorithms] package lets us generate new arrays with random data:<br />
<br />
<haskell><br />
-- 3d array of Ints, bounded between 0 and 255.<br />
> randomishIntArray (Z :. (3::Int) :. (3::Int) :. (3::Int)) 0 255 1<br />
[217,42,130,200,216,254<br />
,67,77,152,85,140,226,179<br />
,71,23,17,152,84,47,17,45<br />
,5,88,245,107,214,136]<br />
</haskell><br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
=== Copying arrays from pointers ===<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Example: Rotating an image with backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40024Numeric Haskell: A Repa Tutorial2011-05-17T19:48:11Z<p>DonStewart: /* Building arrays from vectors */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
=== Building shapes ===<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
=== Working with shapes ===<br />
<br />
That one key operation, '''extent''', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
-- build an array<br />
Repa> let x :: Array DIM3 Int; x = fromList (Z :. (3::Int) :. (3::Int) :. (3::Int)) [1..27]<br />
Repa> :t x<br />
x :: Array DIM3 Int<br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Generating random arrays ===<br />
<br />
The [http://hackage.haskell.org/package/repa-algorithsm repa-algorithms] package lets us generate new arrays with random data:<br />
<br />
<haskell><br />
-- 3d array of Ints, bounded between 0 and 255.<br />
> randomishIntArray (Z :. (3::Int) :. (3::Int) :. (3::Int)) 0 255 1<br />
[217,42,130,200,216,254<br />
,67,77,152,85,140,226,179<br />
,71,23,17,152,84,47,17,45<br />
,5,88,245,107,214,136]<br />
</haskell><br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
=== Copying arrays from pointers ===<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Example: Rotating an image with backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40023Numeric Haskell: A Repa Tutorial2011-05-17T19:41:06Z<p>DonStewart: /* Working with shapes */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
=== Building shapes ===<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
=== Working with shapes ===<br />
<br />
That one key operation, '''extent''', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
-- build an array<br />
Repa> let x :: Array DIM3 Int; x = fromList (Z :. (3::Int) :. (3::Int) :. (3::Int)) [1..27]<br />
Repa> :t x<br />
x :: Array DIM3 Int<br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
=== Copying arrays from pointers ===<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Example: Rotating an image with backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40022Numeric Haskell: A Repa Tutorial2011-05-17T19:40:44Z<p>DonStewart: /* Working with shapes */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
=== Building shapes ===<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
=== Working with shapes ===<br />
<br />
That one key operation, '''extent'', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
-- build an array<br />
Repa> let x :: Array DIM3 Int; x = fromList (Z :. (3::Int) :. (3::Int) :. (3::Int)) [1..27]<br />
Repa> :t x<br />
x :: Array DIM3 Int<br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
=== Copying arrays from pointers ===<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Example: Rotating an image with backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40021Numeric Haskell: A Repa Tutorial2011-05-17T19:36:30Z<p>DonStewart: /* Rotating an image: backpermute */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
=== Building shapes ===<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
=== Working with shapes ===<br />
<br />
That one key operation, '''extent'', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
=== Copying arrays from pointers ===<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Example: Rotating an image with backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40020Numeric Haskell: A Repa Tutorial2011-05-17T19:36:03Z<p>DonStewart: /* Copying arrays from pointers */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
=== Building shapes ===<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
=== Working with shapes ===<br />
<br />
That one key operation, '''extent'', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
=== Copying arrays from pointers ===<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Rotating an image: backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40019Numeric Haskell: A Repa Tutorial2011-05-17T19:35:44Z<p>DonStewart: /* Working with shapes */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
=== Building shapes ===<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
=== Working with shapes ===<br />
<br />
That one key operation, '''extent'', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Copying arrays from pointers ==<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Rotating an image: backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40018Numeric Haskell: A Repa Tutorial2011-05-17T19:35:29Z<p>DonStewart: /* Shapes */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
=== Building shapes ===<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Working with shapes ==<br />
<br />
That one key operation, '''extent'', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Copying arrays from pointers ==<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Rotating an image: backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40017Numeric Haskell: A Repa Tutorial2011-05-17T19:34:35Z<p>DonStewart: /* Shapes */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Working with shapes ==<br />
<br />
That one key operation, '''extent'', gives us many attributes of an array:<br />
<br />
<haskell><br />
-- Extract the shape of the array<br />
extent :: Array sh a -> sh<br />
</haskell><br />
<br />
So, given a 3x3x3 array, of type '''Array DIM3 Int''', we can:<br />
<br />
<haskell><br />
<br />
-- query the extent<br />
Repa> extent x<br />
((Z :. 3) :. 3) :. 3<br />
<br />
-- compute the rank (number of dimensions)<br />
Repa> let sh = extent x<br />
Repa> rank sh<br />
3<br />
<br />
-- compute the size (total number of elements)<br />
> size sh<br />
27<br />
<br />
-- extract the elements of the array as a flat vector<br />
Repa> toVector x<br />
fromList [1,2,3,4,5,6,7,8,9,10<br />
,11,12,13,14,15,16,17,18,19<br />
,20,21,22,23,24,25,26,27] :: Data.Vector.Unboxed.Vector<br />
</haskell><br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Copying arrays from pointers ==<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Rotating an image: backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40016Numeric Haskell: A Repa Tutorial2011-05-17T19:24:52Z<p>DonStewart: /* Quick Tour */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense, rectangular and store elements all of the same type); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors. Open the ghci and import Repa:<br />
<br />
<haskell><br />
Prelude> :m +Data.Array.Repa<br />
Repa> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
Repa> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
Repa> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
Repa> let inputs = [1..10] :: [Double]<br />
Repa> let x = fromList (Z :. (10::Int)) inputs<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
Repa> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
Repa> let x' = fromList (Z :. 10 :: DIM1) inputs<br />
Repa> :t x'<br />
x' :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
Repa> let x2 = fromList (Z :. (5::Int) :. (2::Int)) inputs<br />
Repa> x2<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
Repa> :t x2<br />
x2 :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, as above, if we define it with the type synonym for 2 dimensional Int- indexed arrays, DIM2:<br />
<br />
<haskell><br />
Repa> let x2' = fromList (Z :. 5 :. 2 :: DIM2) inputs<br />
Repa> x2'<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa> :t x2'<br />
x2' :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
Repa> import Data.Vector.Unboxed --recent ghci's support this import syntax<br />
Repa Unboxed> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
Repa Unboxed> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
Repa Unboxed> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
Repa Unboxed> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
Repa> :m +Data.Array.Repa.IO.Matrix<br />
Repa Matrix> let x = fromList (Z :. 5 :. 2 :: DIM2) [1..10]<br />
Repa Matrix> writeMatrixToTextFile "test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
<haskell><br />
Repa Matrix> xx <- readMatrixFromTextFile "test.dat" <br />
Repa Matrix> xx<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
Repa Matrix> :t xx<br />
it :: Array DIM2 Double<br />
</haskell><br />
<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Copying arrays from pointers ==<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays. Since many of the names parallel those in the Prelude, we import Repa qualified:<br />
<br />
Repa> import qualified Data.Array.Repa as Repa<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
Repa> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
Repa> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified as we requested:<br />
<br />
Repa> Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
Repa> extent (Repa.map (^2) x)<br />
(Z :. 3) :. 3<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
The 'x' defined above is a 2D array:<br />
<br />
Repa> extent x<br />
(Z :. 3) :. 3<br />
<br />
If we sum each row:<br />
<br />
Repa> Repa.fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
we get a 1D array instead:<br />
<br />
Repa> extent (Repa.fold (+) 0 x)<br />
Z :. 3<br />
<br />
Similarly, if 'y' is a (3 x 3 x 3) 3D array:<br />
<br />
Repa> let y = fromList ((Z :. 3 :. 3 :. 3) :: DIM3) [1..27]<br />
Repa> y<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0,26.0,27.0]<br />
<br />
we can fold over the inner dimension: <br />
<br />
Repa> Repa.fold (+) 0 y<br />
[6.0,15.0,24.0,33.0,42.0,51.0,60.0,69.0,78.0]<br />
<br />
yielding a 2D (3 x 3) array in place of our 3D (3 x 3 x 3) array<br />
Repa> extent y<br />
((Z :. 3) :. 3) :. 3<br />
Repa> extent (Repa.fold (+) 0 y)<br />
(Z :. 3) :. 3<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
Repa> Repa.zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
Repa> extent it<br />
(Z :. 3) :. 3<br />
<br />
Repa> Repa.zipWith (*) y y <br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,<br />
100.0,121.0,144.0,169.0,196.0,225.0,256.0,289.0,324.0,<br />
361.0,400.0,441.0,484.0,529.0,576.0,625.0,676.0,729.0] --reformatted<br />
Repa> extent it<br />
((Z :. 3) :. 3) :. 3<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Rotating an image: backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40006Numeric Haskell: A Repa Tutorial2011-05-17T04:52:23Z<p>DonStewart: </p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the [http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
x :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, more simply:<br />
<br />
<haskell><br />
x :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Copying arrays from pointers ==<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Rotating an image: backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40005Numeric Haskell: A Repa Tutorial2011-05-17T04:07:36Z<p>DonStewart: </p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the<br />
[http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial], and is based on the [http://www.scipy.org/Tentative_NumPy_Tutorial NumPy tutorial]''.<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
x :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, more simply:<br />
<br />
<haskell><br />
x :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Copying arrays from pointers ==<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Rotating an image: backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40004Numeric Haskell: A Repa Tutorial2011-05-17T03:34:06Z<p>DonStewart: /* Index types and shapes */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the<br />
[http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is the shape of a small 3D array, with shape type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
x :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, more simply:<br />
<br />
<haskell><br />
x :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Copying arrays from pointers ==<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Rotating an image: backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40003Numeric Haskell: A Repa Tutorial2011-05-17T03:32:40Z<p>DonStewart: /* Index types and shapes */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the<br />
[http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Shape types are built somewhat like lists. The constructor '''Z''' corresponds<br />
to a rank zero shape, and is used to mark the end of the list. The ''':.''' constructor adds additional dimensions to the shape. So, for example, the shape:<br />
<br />
(Z:.3 :.2 :.3)<br />
<br />
is a shape of 3d arrays with type<br />
<br />
(Z:.Int:.Int:.Int)<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
x :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, more simply:<br />
<br />
<haskell><br />
x :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Copying arrays from pointers ==<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Rotating an image: backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40002Numeric Haskell: A Repa Tutorial2011-05-17T03:29:58Z<p>DonStewart: /* Index types and shapes */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the<br />
[http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes (technically, a ''heterogeneous snoc list'').<br />
<br />
Index types consist of two parts:<br />
<br />
* a dimension component; and<br />
* an index type<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
x :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, more simply:<br />
<br />
<haskell><br />
x :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Copying arrays from pointers ==<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) <br />
=> (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Rotating an image: backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
<haskell><br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
</haskell><br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=40000Numeric Haskell: A Repa Tutorial2011-05-17T03:03:37Z<p>DonStewart: /* Examples */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the<br />
[http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes.<br />
<br />
Index types consist of two parts:<br />
<br />
* a dimension component; and<br />
* an index type<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
<code><br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
</code><br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
x :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, more simply:<br />
<br />
<haskell><br />
x :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Copying arrays from pointers ==<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) => (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Rotating an image: backpermute ===<br />
<br />
Flip an image upside down:<br />
<br />
<haskell><br />
import System.Environment<br />
import Data.Word<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
main = do<br />
[f] <- getArgs<br />
runIL $ do<br />
v <- readImage f<br />
writeImage ("flip-"++f) (rot180 v)<br />
<br />
rot180 :: Array DIM3 Word8 -> Array DIM3 Word8<br />
rot180 g = backpermute e flop g<br />
where<br />
e@(Z :. x :. y :. _) = extent g<br />
<br />
flop (Z :. i :. j :. k) =<br />
(Z :. x - i - 1 :. y - j - 1 :. k)<br />
</haskell><br />
<br />
Running this:<br />
<br />
$ ghc -O2 --make A.hs<br />
$ ./A haskell.jpg <br />
<br />
Results in:<br />
<br />
http://i.imgur.com/YsGA8.jpg<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=39998Numeric Haskell: A Repa Tutorial2011-05-17T01:42:05Z<p>DonStewart: /* Importing the library */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the<br />
[http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
* [http://hackage.haskell.org/package/repa-devil repa-devil] (image loading)<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes.<br />
<br />
Index types consist of two parts:<br />
<br />
* a dimension component; and<br />
* an index type<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
x :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, more simply:<br />
<br />
<haskell><br />
x :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Copying arrays from pointers ==<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) => (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=39997Numeric Haskell: A Repa Tutorial2011-05-17T01:36:20Z<p>DonStewart: /* Reading arrays from files */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the<br />
[http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes.<br />
<br />
Index types consist of two parts:<br />
<br />
* a dimension component; and<br />
* an index type<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
x :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, more simply:<br />
<br />
<haskell><br />
x :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Copying arrays from pointers ==<br />
<br />
You can also generate new repa arrays by copying data from a pointer, using the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] package. Here is an example, using "copyFromPtrWord8":<br />
<br />
<haskell><br />
import Data.Word<br />
import Foreign.Ptr<br />
<br />
import qualified Data.Vector.Storable as V<br />
import qualified Data.Array.Repa as R<br />
import Data.Array.Repa<br />
import qualified Data.Array.Repa.ByteString as R<br />
<br />
import Data.Array.Repa.IO.DevIL<br />
<br />
i, j, k :: Int <br />
(i, j, k) = (255, 255, 4 {-RGBA-})<br />
<br />
-- 1d vector, filled with pretty colors<br />
v :: V.Vector Word8<br />
v = V.fromList . take (i * j * k) . cycle $ concat<br />
[ [ r, g, b, 255 ]<br />
| r <- [0 .. 255]<br />
, g <- [0 .. 255]<br />
, b <- [0 .. 255]<br />
] <br />
<br />
ptr2repa :: Ptr Word8 -> IO (R.Array R.DIM3 Word8)<br />
ptr2repa p = R.copyFromPtrWord8 (Z :. i :. j :. k) p<br />
<br />
main = do<br />
-- copy our 1d vector to a repa 3d array, via a pointer<br />
r <- V.unsafeWith v ptr2repa<br />
runIL $ writeImage "test.png" r <br />
return ()<br />
</haskell><br />
<br />
This fills a vector, converts it to a pointer, then copies that pointer to a 3d array, before writing the result out as this image:<br />
<br />
http://i.imgur.com/o0Cv2.png<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) => (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=39996Numeric Haskell: A Repa Tutorial2011-05-16T19:48:29Z<p>DonStewart: /* Example: matrix-matrix multiplication */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the<br />
[http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes.<br />
<br />
Index types consist of two parts:<br />
<br />
* a dimension component; and<br />
* an index type<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
x :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, more simply:<br />
<br />
<haskell><br />
x :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) => (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
== Examples ==<br />
<br />
Following are some examples of useful functions that exercise the API.<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=39995Numeric Haskell: A Repa Tutorial2011-05-16T19:45:39Z<p>DonStewart: /* Example: image desaturation */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the<br />
[http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes.<br />
<br />
Index types consist of two parts:<br />
<br />
* a dimension component; and<br />
* an index type<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
x :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, more simply:<br />
<br />
<haskell><br />
x :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) => (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: parallel image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=39994Numeric Haskell: A Repa Tutorial2011-05-16T19:45:24Z<p>DonStewart: /* Example: matrix-matrix multiplication */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the<br />
[http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes.<br />
<br />
Index types consist of two parts:<br />
<br />
* a dimension component; and<br />
* an index type<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
x :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, more simply:<br />
<br />
<haskell><br />
x :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) => (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
=== Example: image desaturation ===<br />
<br />
To convert an image from color to greyscale, we can use the luminosity method to averge RGB pixels into a common grey value, where the average is weighted for human perception of green<br />
<br />
The formula for luminosity is 0.21 R + 0.71 G + 0.07 B.<br />
<br />
We can write a parallel image desaturation tool using repa and the [http://hackage.haskell.org/package/repa-devil repa-devil image library]:<br />
<br />
<haskell><br />
import Data.Array.Repa.IO.DevIL<br />
import Data.Array.Repa hiding ((++))<br />
import Data.Word<br />
import System.Environment<br />
<br />
--<br />
-- Read an image, desaturate, write out with new name.<br />
--<br />
main = do <br />
[f] <- getArgs<br />
runIL $ do<br />
a <- readImage f<br />
let b = traverse a id luminosity <br />
writeImage ("grey-" ++ f) b<br />
</haskell><br />
<br />
And now the luminosity transform itself, which averages the 3 RGB colors based on preceived weight:<br />
<br />
<haskell><br />
--<br />
-- (Parallel) desaturation of an image via the luminosity method.<br />
--<br />
luminosity :: (DIM3 -> Word8) -> DIM3 -> Word8<br />
luminosity _ (Z :. _ :. _ :. 3) = 255 -- alpha channel<br />
luminosity f (Z :. i :. j :. _) = ceiling $ 0.21 * r + 0.71 * g + 0.07 * b<br />
where<br />
r = fromIntegral $ f (Z :. i :. j :. 0)<br />
g = fromIntegral $ f (Z :. i :. j :. 1)<br />
b = fromIntegral $ f (Z :. i :. j :. 2)<br />
</haskell><br />
<br />
And that's it! The result is a parallel image desaturator, when compiled with <br />
<br />
$ ghc -O -threaded -rtsopts --make A.hs -fforce-recomp<br />
<br />
which we can run, to use two cores:<br />
<br />
$ time ./A sunflower.png +RTS -N2 -H <br />
./A sunflower.png +RTS -N2 -H 0.19s user 0.03s system 135% cpu 0.165 total<br />
<br />
Given an image like this:<br />
<br />
http://i.imgur.com/iu4gV.png<br />
<br />
The desaturated result from Haskell:<br />
<br />
http://i.imgur.com/REhA5.png<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=39990Numeric Haskell: A Repa Tutorial2011-05-16T02:45:28Z<p>DonStewart: /* Reading arrays from files */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the<br />
[http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes.<br />
<br />
Index types consist of two parts:<br />
<br />
* a dimension component; and<br />
* an index type<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
x :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, more simply:<br />
<br />
<haskell><br />
x :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
For image IO in many, many formats, use the [http://hackage.haskell.org/package/repa-devil repa-devil] library.<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) => (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Parallel&diff=39920Parallel2011-05-12T04:06:46Z<p>DonStewart: </p>
<hr />
<div>'''Parallelism and Concurrency in Haskell'''<br />
<br />
Haskell supports both pure parallelism and explicit concurrency. How would you like to begin?<br />
<br />
# Speed up your code by making it run on multicore:<br />
#: Start with Control.Parallel ('''par''', '''pseq''') and refine with Strategies<br />
#: ''Find out more about [[Parallelism|parallelism basics]]''<br />
# Manage simultaneous IO actions (eg. multiple connections on a web server)<br />
#: Start with Concurrent Haskell ('''forkIO''', '''MVar''') <br />
#: ''Find out more about [[Concurrency|concurrency basics]]''<br />
# Work with clusters or do distributed programming<br />
#: Learn about concurrency first, then try using [[Applications_and_libraries/Network#Libraries|network protocol libraries]] like HTTP or zeromq.<br />
#: Meanwhile look out for [[Parallel/Research|ongoing research]] into distributed Haskell.<br />
<br />
== Community ==<br />
<br />
* Ask questions on [[Mailing lists|Haskell Cafe]]<br />
* See what [https://groups.google.com/group/parallel-haskell parallel-haskell] researchers and developers are working on<br />
* Follow [http://twitter.com/#!/parallelhaskell @parallelhaskell] on Twitter [[image:Twitter-mini.png]]<br />
* StackOverflow on Haskell [http://stackoverflow.com/questions/tagged/haskell%2bparallel parallelism] and [http://stackoverflow.com/questions/tagged/haskell%2bconcurrency concurrency]<br />
<br />
== News ==<br />
<br />
* 2011-03-17 [https://groups.google.com/d/topic/parallel-haskell/K-IJ2roA59I/discussion Second SISCA Multicore Challenge] - N body problem (registration deadline 18 May) <br />
* 2011-03-31 [http://www.well-typed.com/blog/52 Parallel Haskell Digest 1]<br />
<br />
== Tools ==<br />
<br />
* [http://research.microsoft.com/en-us/projects/threadscope Threadscope] - parallel programs not getting faster? Use the Threadscope debugger and watch sparks fly.<br />
* Comprehensive list of [[Applications_and_libraries/Concurrency_and_parallelism|Parallelism and Concurrency libraries]]<br />
<br />
== Documentation ==<br />
<br />
* [[Parallel/Glossary|Glossary]]<br />
* [[Parallel/Reading|Parallel Haskell reading]]<br />
* [[Parallel/Research|Current research]]<br />
<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Parallel&diff=39919Parallel2011-05-12T04:04:59Z<p>DonStewart: above the fold</p>
<hr />
<div>Parallelism and Concurrency in Haskell<br />
<br />
Haskell supports both pure parallelism and explicit concurrency. How would you like to begin?<br />
<br />
# Speed up your code by making it run on multicore:<br />
#: Start with Control.Parallel ('''par''', '''pseq''') and refine with Strategies<br />
#: ''Find out more about [[Parallelism|parallelism basics]]''<br />
# Manage simultaneous IO actions (eg. multiple connections on a web server)<br />
#: Start with Concurrent Haskell ('''forkIO''', '''MVar''') <br />
#: ''Find out more about [[Concurrency|concurrency basics]]''<br />
# Work with clusters or do distributed programming<br />
#: Learn about concurrency first, then try using [[Applications_and_libraries/Network#Libraries|network protocol libraries]] like HTTP or zeromq.<br />
#: Meanwhile look out for [[Parallel/Research|ongoing research]] into distributed Haskell.<br />
<br />
== Community ==<br />
<br />
* Ask questions on [[Mailing lists|Haskell Cafe]]<br />
* See what [https://groups.google.com/group/parallel-haskell parallel-haskell] researchers and developers are working on<br />
* Follow [http://twitter.com/#!/parallelhaskell @parallelhaskell] on Twitter [[image:Twitter-mini.png]]<br />
* StackOverflow on Haskell [http://stackoverflow.com/questions/tagged/haskell%2bparallel parallelism] and [http://stackoverflow.com/questions/tagged/haskell%2bconcurrency concurrency]<br />
<br />
== News ==<br />
<br />
* 2011-03-17 [https://groups.google.com/d/topic/parallel-haskell/K-IJ2roA59I/discussion Second SISCA Multicore Challenge] - N body problem (registration deadline 18 May) <br />
* 2011-03-31 [http://www.well-typed.com/blog/52 Parallel Haskell Digest 1]<br />
<br />
== Tools ==<br />
<br />
* [http://research.microsoft.com/en-us/projects/threadscope Threadscope] - parallel programs not getting faster? Use the Threadscope debugger and watch sparks fly.<br />
* Comprehensive list of [[Applications_and_libraries/Concurrency_and_parallelism|Parallelism and Concurrency libraries]]<br />
<br />
== Documentation ==<br />
<br />
* [[Parallel/Glossary|Glossary]]<br />
* [[Parallel/Reading|Parallel Haskell reading]]<br />
* [[Parallel/Research|Current research]]<br />
<br />
[[Category:Parallel]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=39918Numeric Haskell: A Repa Tutorial2011-05-11T23:02:58Z<p>DonStewart: /* Indexing arrays */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the<br />
[http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes.<br />
<br />
Index types consist of two parts:<br />
<br />
* a dimension component; and<br />
* an index type<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
x :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, more simply:<br />
<br />
<haskell><br />
x :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
<haskell><br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
</haskell><br />
<br />
So:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0<br />
</haskell><br />
<br />
Note that we can't give just a bare literal as the shape, even for one-dimensional arrays, :<br />
<br />
<haskell><br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
</haskell><br />
<br />
as the Z type isn't in the Num class, and Haskell's numeric literals are overloaded.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
<haskell><br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
</haskell><br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
<haskell><br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
</haskell><br />
<br />
An example:<br />
<br />
<haskell><br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
</haskell><br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) => (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=39917Numeric Haskell: A Repa Tutorial2011-05-11T22:57:48Z<p>DonStewart: /* Generating arrays */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the<br />
[http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes.<br />
<br />
Index types consist of two parts:<br />
<br />
* a dimension component; and<br />
* an index type<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways, and we always begin by importing the <code>Data.Array.Repa</code> module:<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
They may be constructed from lists, for example. Here is a one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
The type of `x` is inferred as:<br />
<br />
<haskell><br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
</haskell><br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
<haskell><br />
x :: Array DIM1 Double<br />
</haskell><br />
<br />
The same data may also be treated as a two dimensional array, by changing the shape parameter:<br />
<br />
<haskell><br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
</haskell><br />
<br />
which has the type:<br />
<br />
<haskell><br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
or, more simply:<br />
<br />
<haskell><br />
x :: Array DIM2 Double<br />
</haskell><br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors, from the 'vector' package:<br />
<br />
<haskell><br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
</haskell><br />
<br />
New arrays are built by applying a shape to the vector. For example:<br />
<br />
<haskell><br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
</haskell><br />
<br />
is a one-dimensional array of doubles. As usual, we can also impose other<br />
shapes:<br />
<br />
<haskell><br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
</haskell><br />
<br />
to create a 3x3 array.<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
with other formats rapidly appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
This will result in a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Reads this .bmp image:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8, which can be further processed.<br />
<br />
''Note: at the time of writing, there are no binary instances for repa arrays''<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
<br />
So:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0 <br />
<br />
Note that we can't, even for one-dimensional arrays, give just a bare literal as the shape:<br />
<br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
<br />
as the Z type isn't in the Num class.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
<br />
An example:<br />
<br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) => (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]</div>DonStewarthttps://wiki.haskell.org/index.php?title=Numeric_Haskell:_A_Repa_Tutorial&diff=39916Numeric Haskell: A Repa Tutorial2011-05-11T22:50:25Z<p>DonStewart: /* Shapes */</p>
<hr />
<div>[http://hackage.haskell.org/package/repa Repa] is a Haskell library for<br />
high performance, regular, multi-dimensional parallel arrays. All<br />
numeric data is stored unboxed and functions written with the Repa<br />
combinators are automatically parallel (provided you supply "+RTS -N" on<br />
the command line when running the program).<br />
<br />
[[Image:Grid.png|right]]<br />
<br />
This document provides a tutorial on array programming in Haskell using the repa package.<br />
<br />
''Note: a companion tutorial to this is provided as the<br />
[http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Vector_Tutorial vector tutorial].''<br />
<br />
= Quick Tour =<br />
<br />
Repa (REgular PArallel arrays) is an advanced, multi-dimensional parallel arrays library for Haskell, with a number of distinct capabilities:<br />
<br />
* The arrays are "regular" (i.e. dense and rectangular); and<br />
* Functions may be written that are polymorphic in the shape of the array;<br />
* Many operations on arrays are accomplished by changing only the shape of the array (without copying elements);<br />
* The library will automatically parallelize operations over arrays.<br />
<br />
This is a quick start guide for the package. For further information, consult:<br />
<br />
* [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa.html The Haddock Documentation]<br />
* [http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf Regular, Shape-polymorphic, Parallel Arrays in Haskell].<br />
* [http://www.cse.unsw.edu.au/~benl/papers/stencil/stencil-icfp2011-sub.pdf Efficient Parallel Stencil Convolution in Haskell]<br />
<br />
== Importing the library ==<br />
<br />
Download the `repa` package:<br />
<br />
$ cabal install repa<br />
<br />
and import it qualified:<br />
<br />
import qualified Data.Array.Repa as R<br />
<br />
The library needs to be imported qualified as it shares the same function names as list operations in the Prelude.<br />
<br />
''Note: Operations that involve writing new index types for Repa arrays will require the '-XTypeOperators' language extension.''<br />
<br />
For non-core functionality, a number of related packages are available:<br />
<br />
* [http://hackage.haskell.org/package/repa-bytestring repa-bytestring]<br />
* [http://hackage.haskell.org/package/repa-io repa-io]<br />
* [http://hackage.haskell.org/package/repa-algorithms repa-algorithms]<br />
<br />
and example algorithms in:<br />
<br />
* [http://hackage.haskell.org/package/repa-examples repa-examples]<br />
<br />
== Index types and shapes ==<br />
<br />
Before we can get started manipulating arrays, we need a grasp of repa's notion of array shape. Much like the classic 'array' library in Haskell, repa-based arrays are parameterized via a type which determines the dimension of the array, and the type of its index. However, while classic arrays take tuples to represent multiple dimensions, Repa arrays use a [http://hackage.haskell.org/packages/archive/repa/2.0.0.3/doc/html/Data-Array-Repa-Shape.html#t:Shape richer type language] for describing multi-dimensional array indices and shapes.<br />
<br />
Index types consist of two parts:<br />
<br />
* a dimension component; and<br />
* an index type<br />
<br />
The most common dimensions are given by the shorthand names:<br />
<br />
<haskell><br />
type DIM0 = Z<br />
type DIM1 = DIM0 :. Int<br />
type DIM2 = DIM1 :. Int<br />
type DIM3 = DIM2 :. Int<br />
type DIM4 = DIM3 :. Int<br />
type DIM5 = DIM4 :. Int<br />
</haskell><br />
<br />
thus,<br />
<br />
<haskell><br />
Array DIM2 Double<br />
</haskell><br />
<br />
is the type of a two-dimensional array of doubles, indexed via `Int` keys, while<br />
<br />
<haskell><br />
Array Z Double<br />
</haskell><br />
<br />
is a zero-dimension object (i.e. a point) holding a Double.<br />
<br />
Many operations over arrays are polymorphic in the shape / dimension<br />
component. Others require operating on the shape itself, rather than<br />
the array. A typeclass, <code>Shape</code>, lets us operate uniformally<br />
over arrays with different shape.<br />
<br />
== Shapes ==<br />
<br />
To build values of `shape` type, we can use the <code>Z</code> and <code>:.</code> constructors:<br />
<br />
<haskell><br />
> Z -- the zero-dimension<br />
Z<br />
</haskell><br />
<br />
For arrays of non-zero dimension, we must give a size. ''Note:'' a common error is to leave off the type of the size.<br />
<br />
<haskell><br />
> :t Z :. 10<br />
Z :. 10 :: Num head => Z :. head<br />
</haskell><br />
<br />
leading to annoying type errors about unresolved instances, such as:<br />
<br />
No instance for (Shape (Z :. head0))<br />
<br />
To select the correct instance, you will need to annotate the size literals with their concrete type:<br />
<br />
<haskell><br />
> :t Z :. (10 :: Int)<br />
Z :. (10 :: Int) :: Z :. Int<br />
</haskell><br />
<br />
is the shape of 1D arrays of length 10, indexed via Ints.<br />
<br />
Given an array, you can always find its shape by calling <code>extent</code>.<br />
<br />
Additional convenience types for selecting particular parts of a shape are also provided (<code>All, Any, Slice</code> etc.) which are covered later in the tutorial.<br />
<br />
== Generating arrays ==<br />
<br />
New repa arrays ("arrays" from here on) can be generated in many ways:<br />
<br />
$ ghci<br />
GHCi, version 7.0.3: http://www.haskell.org/ghc/ :? for help<br />
Loading package ghc-prim ... linking ... done.<br />
Loading package integer-gmp ... linking ... done.<br />
Loading package base ... linking ... done.<br />
Loading package ffi-1.0 ... linking ... done.<br />
Prelude> :m + Data.Array.Repa<br />
<br />
They may be constructed from lists:<br />
<br />
A one dimensional array of length 10, here, given the shape `(Z :. 10)`:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
<br />
The type of `x` is inferred as:<br />
<br />
> :t x<br />
x :: Array (Z :. Int) Double<br />
<br />
which we can read as "an array of dimension 1, indexed via Int keys, holding elements of type Double"<br />
<br />
We could also have written the type as:<br />
<br />
x :: Array DIM1 Double<br />
<br />
The same data may also be treated as a two dimensional array:<br />
<br />
> let x = fromList (Z :. (5::Int) :. (2::Int)) [1..10]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]<br />
<br />
which would have the type:<br />
<br />
x :: Array ((Z :. Int) :. Int) Double<br />
<br />
or<br />
x :: Array DIM2 Double<br />
<br />
=== Building arrays from vectors ===<br />
<br />
It is also possible to build arrays from unboxed vectors:<br />
<br />
fromVector :: Shape sh => sh -> Vector a -> Array sh a<br />
<br />
by applying a shape to a vector.<br />
<br />
import Data.Vector.Unboxed<br />
<br />
> let x = fromVector (Z :. (10::Int)) (enumFromN 0 10)<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
is a one-dimensional array of doubles, but we can also impose other<br />
shapes:<br />
<br />
> let x = fromVector (Z :. (3::Int) :. (3::Int)) (enumFromN 0 9)<br />
> x<br />
[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]<br />
> :t x<br />
x :: Array ((Z :. Int) :. Int) Double<br />
<br />
=== Reading arrays from files ===<br />
<br />
Using the [http://hackage.haskell.org/package/repa-io repa-io] package, arrays may be written and read from files in a number of formats:<br />
<br />
* as BMP files; and<br />
* in a number of text formats.<br />
<br />
With other formats appearing. For the special case of arrays of Word8 values, the [http://hackage.haskell.org/package/repa-bytestring repa-bytestring] library supports generating bytestrings in memory.<br />
<br />
An example: to write an 2D array to an ascii file:<br />
<br />
<haskell><br />
writeMatrixToTextFile "/tmp/test.dat" x<br />
</haskell><br />
<br />
Yields a file containing:<br />
<br />
MATRIX<br />
2 5<br />
1.0<br />
2.0<br />
3.0<br />
4.0<br />
5.0<br />
6.0<br />
7.0<br />
8.0<br />
9.0<br />
10.0<br />
<br />
In turn, this file may be read back in via <code>readMatrixFromTextFile</code>.<br />
<br />
To process [http://en.wikipedia.org/wiki/BMP_file_format .bmp files], use [http://hackage.haskell.org/packages/archive/repa-io/2.0.0.3/doc/html/Data-Array-Repa-IO-BMP.html Data.Array.Repa.IO.BMP], as follows (currently reading only works for 24 bit .bmp):<br />
<br />
<haskell><br />
Data.Array.Repa.IO.BMP> x <- readImageFromBMP "/tmp/test24.bmp"<br />
</haskell><br />
<br />
Results in this being read:<br />
<br />
http://i.imgur.com/T4ttx.png<br />
<br />
as a 3D array of Word8.<br />
<br />
== Indexing arrays ==<br />
<br />
To access elements in repa arrays, you provide an array and a shape, to access the element:<br />
<br />
(!) :: (Shape sh, Elt a) => Array sh a -> sh -> a<br />
<br />
So:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x ! (Z :. 2)<br />
3.0 <br />
<br />
Note that we can't, even for one-dimensional arrays, give just a bare literal as the shape:<br />
<br />
> x ! 2<br />
<br />
No instance for (Num (Z :. Int))<br />
arising from the literal `2'<br />
<br />
as the Z type isn't in the Num class.<br />
<br />
What if the index is out of bounds, though?<br />
<br />
> x ! (Z :. 11)<br />
*** Exception: ./Data/Vector/Generic.hs:222 ((!)): index out of bounds (11,10)<br />
<br />
an exception is thrown. An altnerative is to indexing functions that return a Maybe:<br />
<br />
(!?) :: (Shape sh, Elt a) => Array sh a -> sh -> Maybe a<br />
<br />
An example:<br />
<br />
> x !? (Z :. 9)<br />
Just 10.0<br />
<br />
> x !? (Z :. 11)<br />
Nothing<br />
<br />
== Operations on arrays ==<br />
<br />
Besides indexing, there are many regular, list-like operations on arrays.<br />
<br />
=== Maps, zips, filters and folds ===<br />
<br />
We can map over multi-dimensional arrays:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
since `map` conflicts with the definition in the Prelude, we have to use it qualified:<br />
<br />
> Data.Array.Repa.map (^2) x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
Maps leave the dimension unchanged.<br />
<br />
Folding reduces the inner dimension of the array.<br />
<br />
fold :: (Shape sh, Elt a) => (a -> a -> a) -> a -> Array (sh :. Int) a -> Array sh a<br />
<br />
So if 'x' is a 3D array:<br />
<br />
> let x = fromList (Z :. (3::Int) :. (3::Int)) [1..9]<br />
> x<br />
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0]<br />
<br />
We can sum each row, to yield a 2D array:<br />
<br />
> fold (+) 0 x<br />
[6.0,15.0,24.0]<br />
<br />
Two arrays may be combined via <code>zipWith</code>:<br />
<br />
zipWith :: (Shape sh, Elt b, Elt c, Elt a) =><br />
(a -> b -> c) -> Array sh a -> Array sh b -> Array sh c<br />
<br />
an example:<br />
<br />
> zipWith (*) x x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0]<br />
<br />
=== Numeric operations: negation, addition, subtraction, multiplication ===<br />
<br />
Repa arrays are instances of the <code>Num</code>. This means that<br />
operations on numerical elements are lifted automagically onto arrays of<br />
such elements. For example, <code>(+)</code> on two double values corresponds to<br />
element-wise addition, <code>(+)</code>, of the two arrays of doubles:<br />
<br />
> let x = fromList (Z :. (10::Int)) [1..10]<br />
> x + x<br />
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]<br />
<br />
Other operations from the Num class work just as well:<br />
<br />
> -x<br />
[-1.0,-2.0,-3.0,-4.0,-5.0,-6.0,-7.0,-8.0,-9.0,-10.0]<br />
<br />
> x ^ 3<br />
[1.0,8.0,27.0,64.0,125.0,216.0,343.0,512.0,729.0,1000.0]<br />
<br />
> x * x<br />
[1.0,4.0,9.0,16.0,25.0,36.0,49.0,64.0,81.0,100.0]<br />
<br />
== Changing the shape of an array ==<br />
<br />
One of the main advantages of repa-style arrays over other arrays in<br />
Haskell is the ability to reshape data without copying. This is achieved<br />
via *index-space transformations*. <br />
<br />
An example: transposing a 2D array (this example taken from the repa<br />
paper). First, the type of the transformation:<br />
<br />
transpose2D :: Elt e => Array DIM2 e -> Array DIM2 e <br />
<br />
Note that this transform will work on DIM2 arrays holding any elements.<br />
Now, to swap rows and columns, we have to modify the shape:<br />
<br />
transpose2D a = backpermute (swap e) swap a<br />
where<br />
e = extent a<br />
swap (Z :. i :. j) = Z :. j :. i<br />
<br />
The swap function reorders the index space of the array.<br />
To do this, we extract the current shape of the array, and write a function<br />
that maps the index space from the old array to the new array. That index space function<br />
is then passed to backpermute which actually constructs the new<br />
array from the old one.<br />
<br />
backpermute generates a new array from an old, when given the new shape, and a<br />
function that translates between the index space of each array (i.e. a shape<br />
transformer).<br />
<br />
backpermute<br />
:: (Shape sh, Shape sh', Elt a) <br />
=> sh' <br />
-> (sh' -> sh) <br />
-> Array sh a <br />
-> Array sh' a<br />
<br />
Note that the array created is not actually evaluated (we only modified the index space of the array).<br />
<br />
Transposition is such a common operation that it is provided by the<br />
library:<br />
<br />
transpose :: (Shape sh, Elt a)<br />
=> Array ((sh :. Int) :. Int) a -> Array ((sh :. Int) :. Int)<br />
<br />
the type indicate that it works on the lowest two dimensions of the<br />
array.<br />
<br />
Other operations on index spaces include taking slices and joining<br />
arrays into larger ones.<br />
<br />
=== Example: matrix-matrix multiplication ===<br />
<br />
A more advanced example from the Repa paper: matrix-matrix multiplication: the result of<br />
matrix multiplication is a matrix whose elements are found by<br />
multiplying the elements of each row from the first matrix by the<br />
associated elements of the same column from the second matrix and<br />
summing the result.<br />
<br />
if <math>A=\begin{bmatrix}a&b\\c&d\end{bmatrix}</math> and <math>B=\begin{bmatrix}e&f\\g&h\end{bmatrix}</math><br />
<br />
then <br />
<br />
<math>AB=\begin{bmatrix}a&b\\c&d\end{bmatrix}\begin{bmatrix}e&f\\g&h\end{bmatrix}=\begin{bmatrix}ae+bg&af+bh\\ce+dg&cf+dh\end{bmatrix}</math><br />
<br />
So we take two, 2D arrays and generate a new array, using our transpose<br />
function from earlier:<br />
<br />
mmMult :: (Num e, Elt e)<br />
=> Array DIM2 e<br />
-> Array DIM2 e<br />
-> Array DIM2 e<br />
<br />
mmMult a b = sum (zipWith (*) aRepl bRepl)<br />
where<br />
t = transpose2D b<br />
aRepl = extend (Z :.All :.colsB :.All) a<br />
bRepl = extend (Z :.rowsA :.All :.All) t<br />
(Z :.colsA :.rowsA) = extent a<br />
(Z :.colsB :.rowsB) = extent b<br />
<br />
The idea is to expand both 2D argument arrays into 3D arrays by<br />
replicating them across a new axis. The front face of the cuboid that<br />
results represents the array <code>a</code>, which we replicate as often<br />
as <code>b</code> has columns <code>(colsB)</code>, producing<br />
<code>aRepl</code>.<br />
<br />
The top face represents <code>t</code> (the transposed b), which we<br />
replicate as often as a has rows <code>(rowsA)</code>, producing<br />
<code>bRepl,</code>. The two replicated arrays have the same extent,<br />
which corresponds to the index space of matrix multiplication<br />
<br />
Optimized implementations of this function are available in the<br />
repa-algorithms package.<br />
<br />
<br />
[[Category:Libraries]]<br />
[[Category:Packages]]<br />
[[Category:Performance]]<br />
[[Category:Tutorials]]</div>DonStewart