No items found.
For demonstration purposes, we're going to build a simple bibliographic resources management tool – BiblioteK. Although BiblioteK aims to be a big name in the industry (we're looking at you, Zotero), but that's a mission for you if you want to help out. For our demo, though, it'll have limited functionality. It'll compress a humble CLI interface to manage and store resources on your computer.
The usability then it'll look like this:
# create a resource
btek resources create --kind book --title "Functional Programming in Scala" --authors "Michael Pilquist, Runar Bjarnason, Paul Chiusano" --category Programming --published 2024
# list resources
btek resources list
# Programming
# [8u4xhd] Functional Programming in Scala (Michael Pilquist, Runar Bjarnason, Paul Chiusano, 2024)
# delete
btek resources remove 8u4xhd
To manage material associated with a given resource, let's say a video or a PDF, the user will have to type something similar to this:
# material
btek material add 8u4xhdIf --name 'My PDF' download.pdf
btek material add 8u4xhdIf video.mp4
btek material list
# Programming
# [8u4xhdIf] Functional Programming in Scala
# - [Yif39s3f] My PDF.pdf
# - [123dfsl9] video.mp4
# Deletes only the specified material.
btek material remove Yif39s3f
# Removes the specified material under
btek material remove 8u4xhdIf video.mp4
# Removes all material associated with the resource.
btek material remove 8u4xhdIfZIO leverages the pattern of programming upon interfaces, selects implementations for those interfaces, and then builds the layers your application needs. We'll explore this in decent depth through the post.
For our sample application, we'll have an architecture that resembles something like the following:
The models of our application are almost what you'd expect from a simple application, with the difference that I love using the newtype pattern, which consists of having important types like the id of the model having its own type that avoids being automatically conflated with its underlying type. For example, for the Resource.Id type, its underlying type is a UUID, but not any UUID is a Resource's id, and a Resource's id is not any UUID – we'd have to convert them manually so is always obvious which is which.
See here what I am talking about:
import monix.newtypes.NewtypeValidated
enum ResourceKind:
case Book
case Video
final case class Resource(
id: Resource.Id,
title: String,
kind: ResourceKind,
category: Option[Category],
publicationYear: Option[Short],
authors: Set[Author],
)
object Resource:
type Id = Id.Type
object Id extends NewtypeValidated[String]:
def apply(value: String): Either[BuildFailure[Type], Type] =
if value.length() == 8 then Right(unsafeCoerce(value))
else Left(BuildFailure("String wasn't 8 characters long"))
def make: UIO[Id] =
makeShortUUID.map: s =>
Id(s).getOrElse(throw new RuntimeException("unreachable"))
end Resource
final case class ResourceRow(
id: Resource.Id,
title: String,
kind: ResourceKind,
publicationYear: Option[Short],
categoryName: Option[Category.Name] = None,
)
I'm using Monix's Newtype library, which gives us a simple wrapper around an opaque type, with an apply function to create a newtype from its underlying type, and a newtype#value method to go the other way around. This wrapper can be either a NewtypeWrapper or a NewtypeValidated, the difference being that the latter expects an Either as its return type for validation.
Here's a glimpse of all our models, which apart from the newtype thing, are quite simple:
final case class Author(id: Author.Id, name: String)
object Author:
type Id = Id.Type
object Id extends NewtypeWrapped[UUID]
final case class Category(name: Category.Name)
object Category:
type Name = Name.Type
object Name extends NewtypeWrapped[String]
final case class Material(id: Material.Id, name: Material.Name, resourceId: Resource.Id)
object Material:
type Id = Id.Type
object Id extends NewtypeWrapped[UUID]
type Name = Name.Type
object Name extends NewtypeWrapped[String]
This pattern is not exclusive to ZIO. I've used it with Cats Effect too. It consists in modelling all the errors your application might have into specific case classes with a common inheritance. I think this pattern has special importance because of the +E type in our ZIO monad, because of its covariance, we could operate upon different kinds of error and the compiler will set the error type as their common ancestor, which normally is Throwable. By modelling the errors, we want to get rid of that Throwable and make a DomainError the common ancestor of all errors in our app, so we can handle them more gracefully on the edges.
package dev.zio.content.bibliotek.domain
import java.io.IOException
import java.sql.SQLException
import scala.reflect.runtime.universe.*
import scala.util.control.NoStackTrace
sealed trait DomainError(val message: String) extends NoStackTrace
object DomainError:
final case class RepositoryError(exception: SQLException)
extends DomainError(message = exception.getMessage())
final case class IOError(exception: IOException)
extends DomainError(message = exception.getMessage())
case class NotFoundError[A](what: A) extends DomainError(message = s"$what was not found")Our approach to configuration falls a bit away from what's commonly used in applications. Normally, you'd like to configure your, let's say, HTTP server, from a file inside the repository directory, but ours ought to be configured by a user and persisted on their computer. This is why we need a config to know where the config should reside. For this purpose, the user could define a location by setting an environment variable, but, if not set, the default location will be the user's home plus the .bibliotek directory.
Set by the env varBIBLIOTEK_CONFIG_DIR.default:$HOME/.bibliotek
For more complex CLI applications, this directory would in turn have a configuration file to parse. For this application, though, configuring the folder is enough, because all we want is a place to put things in there. That's why we're not using something like zio-config, but I beg you to check its documentation for a more robust configuration platform.
Remember that at last, all you want to do is read a file and parse it into a case class – not for this one though.
ZIO comes with a first-class configuration support. The function ZIO.config[A] either receives a zio.Config as an implicit or explicit parameter. The configuration object can be constructed using zio.Config's static constructors. For example, zio.Config.string("CONFIG_SOURCE") describes that we want a string from a configuration source. Configuration sources for zio.Config can be configured as a runtime layer at the application's bootstrap. Here's more information about it. For now, suffice it to say that by default the config provider will look at environment variables and system properties, which is exactly what we wanna do here.
The directory we're going to use for storage will be either one defined by the user by setting an environment variable or, in case is not defined, the user.home system property, with a .bibliotek directory appended.
given config: Config[AppConfig] =
Config
.string("BIBLIOTEK_CONFIG_DIR")
.orElse(Config.string("user.home"))
.map(Path(_) / ".bibliotek")
.map(AppConfig.apply)
ZIO.config[A] is what brings it into existence, i.e. actually get the values:
def layer = ZLayer.fromZIO(ZIO.config(config))
We could just call ZIO.config wherever we want to use our config, but having it as a layer adds flexibility in how we construct our config – using a library like zio-config won't let us have a zio.Config instance right away, but only after some effectful operation.
I remember a talk from the creator of Clojure in which he stated that almost every application has to be able to persist data in a database, and this one is no exception.
Creating the application for this post I almost fell into the rabbit hole that is file-based storage, using something like JSON to encode it, but even if it seems simpler at first glance, there are a lot of edge cases that you'd have to deal with by yourself and I'm not that smart nor industrious. Hence, I decided to not reinvent the wheel, and use a database system.
As we're going to store everything in the user's computer, it quickly came to my mind SQLite, which lets us use our SQL knowledge, while keeping everything in a tight file, and doesn't need a running process (like Postgres) to work. The choice is made.
Quill is the official ZIO library for dealing with SQL. It uses a DSL to build queries at compile time. It has a lot of flexibility, but the basic features were enough for building this application.
We'll need to add a dependency on a driver for Sqlite.
"org.xerial" % "sqlite-jdbc" % sqliteVersion
Only one thing fell out of the normal: As we modelled some of our data types using the newtype pattern (see the models section), we have to define custom encoders for them in the following sections.
For Quill to know how to access our database, we need a java.sql.DataSource. This source, for Sqlite, could be constructed from the import org.sqlite.SQLiteDataSource, which comes from our driver. With that, and the file in which our database will be stored, we can have our instance.
def makeSqliteDataSource(dbFile: Path): UIO[SQLiteDataSource] =
val ds = new SQLiteDataSource()
ZIO.succeed(ds.setUrl(s"jdbc:sqlite:$dbFile")).as(ds)
I used ZIO.succeed because of the mutation, although it doesn't really matter in this case.
This function could be used to build our data source from our configuration.
def dataSourceLayer: RLayer[AppConfig, DataSource] =
ZLayer.fromZIO:
for
cfg <- ZIO.service[AppConfig]
filePath = cfg.configDir / "bibliotek.db"
ds <- makeSqliteDataSource(filePath)
_ <- ZIO.attemptBlocking(cfg.configDir.toFile.mkdir()).debug("Created database directory") // (1)
_ <- ZIO.attemptBlocking(filePath.toFile.createNewFile()).debug("Created database file") // (2)
yield ds
Once we have our data source, I proceed to create the actual directory (1) and file (2) we specified at the data source.
java.io.File#mkdir will only create one level deep in the directory, so the user has to specify a directory parent that exists at the configuration.
To bootstrap our database, I thought of a very simple setup function, which loads up an init.sql file from our resources, and split the statements using a semicolon (I know is dangerous, but since we know exactly the contents of init.sql, I see no problem) into individual statements, and execute each using ZIO.foreach, which traverses (this is how this pattern is called) a list and runs an effect on each element, accumulating the results back in a list.
def setupDatabase: RIO[Quill.Sqlite[SnakeCase], Unit] =
for
quill <- ZIO.service[Quill.Sqlite[SnakeCase]] // 1
sql <- ZIO.attemptBlocking(Source.fromResource("init.sql").mkString)
statements = sql.split(";").dropRight(1).map(_ ++ ";")
_ <- ZIO.foreach(statements)(s => quill.executeAction(s)(io.getquill.context.ExecutionInfo.unknown, ()))
yield ()
Notice how in (1) we ask for a Quill.Sqlite[SnakeCase] service. Where is coming from? So far, we don't care. ZIO will take care of calling out for it when using this function. This is what the first type parameter of RIO stands for. It's storing the requirements for this function to work, but doesn't make the assumption of where it is coming from.
To get a Quill.Sqlite[SnakeCase] instance, we have our function that returns a DataSource, and that will serve as input for:
val sqliteLayer: RIO[DataSource, Quill.Sqlite[SnakeCase]] = Quill.Sqlite.fromNamingStrategy(SnakeCase)
With this, we have our database setup.
This is something I didn't like about Quill, but everything has to live within a context – not of a coconut, but of a database system. For this reason, our encoders and decoders have to live in a trait that has a non-implemented context variable, which then we need to import its contents.
Let's see the resources schema as an example:
import io.getquill.{MappedEncoding, SnakeCase}
trait ResourceSchema:
given resourceIdEncoder: MappedEncoding[UUID, Resource.Id] = MappedEncoding[UUID, Resource.Id](Resource.Id(_))
given resourceIdDecoder: MappedEncoding[Resource.Id, UUID] = MappedEncoding[Resource.Id, UUID](_.value)
given resourceKindEncoder: MappedEncoding[ResourceKind, String] =
MappedEncoding[ResourceKind, String](_.toString.toLowerCase)
given resourceKindDecoder: MappedEncoding[String, ResourceKind] =
MappedEncoding[String, ResourceKind](s => ResourceKind.valueOf(s.capitalize))
end ResourceSchema
There used to be another way to define encoders and decoders, but it seems to be removed. It consisted in using one of the predefine encoders/decoders, and then usecontramapormap, respectively, to get a new type from them. This approach, though, needs of a context to be imported, and this could be done by define an abstractQuill.Contextthat needs to be implemented when using the schema.
io.getquill.MappedEncoding is our mate here. We have to define a two-directional encoding (that'll also be used as a decoder) for each of the types that fall outside the common types for which Quill has already implementations. For our Resource.Id newtype we used a simple apply/value map and contramapping. For our ResourceKind, I have followed a rather naive implementation that simply lower-case or capitalizes the string we get from the database to make it a proper ResourceKind
The rest of the schemas (i.e. Author's and Category's) follow the exact same principles, so I won't bother you with repeated code, but you can check it out at the post's code repository.
Sticking to the layer pattern to build applications. We create interfaces and use them instead of the actual implementation, implementation which has some dependencies in their primary constructor.
// Interface
trait ResourcesRepo:
def listResources: Stream[RepositoryError, Resource]
// Implementation
case class ResourcesRepoLive(protected val quill: Quill.Sqlite[SnakeCase])
extends ResourcesRepo
with ResourceSchema:
def listResources: Stream[RepositoryError, Resource] = ???
See how many more interconnections have the implementation, for example, the schemas and the quill instance. These interconnections translate to dependencies in our code. We delay the moment to think about how we're gonna build those dependencies – or even think about which ones are we going to need, for that matter.
In other languages, this introduces a new whole world of complexity and uncertainty: Dependency Injection. Here on the other hand, you have
Our Quill.Sqlite[?] will be the cornerstone of this layer. With this instance and everything that exports io.getquill we can create query descriptions that will be translated, at compile time, into SQL queries.
Let's explore our main repository: ResourcesRepo.
In ZIO, is customary to implement these as a class that extends this interface plus the suffix live. We'd need to include our schema too, so when we're building queries quill can know which decoder/encoder to use for a type it doesn't know, like our newtypes we introduced earlier in our models.
class ResourcesRepoLive(protected val quill: Quill.Sqlite[SnakeCase])
extends ResourcesRepo
with ResourceSchema
with CategorySchema
with AuthorSchema:
import quill.*
To access a table in our queries, we need a querySchema that maps to a table name.
class ResourcesRepoLive(protected val quill: Quill.Sqlite[SnakeCase]):
// ...
inline def resources = quote:
querySchema[ResourceRow]("resources")
With this, we can start writing our first, which is going to be def listResources: Stream[RepositoryError, ResourceRow]. Here we have our first interesting feature of Quill and something I find nice resulting from having zio.stream as a first-class citizen within ZIO.
// ...
override def listResources: Stream[RepositoryError, ResourceRow] =
val expr = quote(resources) // (1)
stream(expr).refineOrDie: // (2)
case e: SQLException => RepositoryError(e)
The refineOrDie combinator receives a PartialFunction in which we, well, refine the matched errors into a new one, or die if it fails with any of the unmatched errors.To get a stream from a query is trivial. First, we build the expression (1). Then we run it (2). As we are listing every single resource, we don't need filters. Having a stream instead of a list gives us the flexibility to express different list processings without having all load into memory all at once. We can build pagination from it if necessary, for example, without having to change our query.
For adding a resource, will need to execute an action rather than query something. This is accomplished by calling an action method on our query schema, in this case, insertValue.
override def addResource(data: ResourceData): IO[RepositoryError, ResourceRow] =
val effect =
for
resource <- ResourceRow.fromData(data) // (1)
_ <- run(quote(resources.insertValue(lift(resource)))) // (2)
yield resource
effect
.retryN(2) // (3) Retry is here because the id might be already picked.
.mapError(RepositoryError(_))
end addResource
ResourceRow.fromData(data) is a helper function that returns an effect, i.e. a ZIO value, so it needs to be sequenced with the runing of the quoted expression. This is because creating a random id is a side-effect, meaning something that is not referentially transparent. Notice how at (2) we need to lift the resource value. This is a requirement from Quill's quotations.
The other interesting fact about this method is that we're going to try (3) three times the effect of creating a ResourceRow (and its id) and running the insert action. I did this to illustrate how easy it is for an effect-based program to retry the same action in case something went wrong, in this case, the possibility that our random id might already have been picked, a small possibility, but something we won't want our users to worry about.
Our use cases include 'removing' a resource. This one is interesting because although we don't want to remove the authors associated with it (because they might be associated with another resource), we want to remove the associations to the resource.
Many to many associations require their own table, and therefore their own case class. In this application, they have this signature.
final case class ResourceAuthor(resourceId: Resource.Id, authorId: Author.Id)This means we also need another querySchema for our new table: inline def resourceAuthors = quote(querySchema[ResourceAuthor]("resource_authors")) override def removeResource(resourceId: Resource.Id): IO[DomainError, Unit] =
val expr = quote: // (1)
resources.filter(_.id == lift(resourceId)).delete
val deleteAuthorsAssocs = quote: // (2)
resourceAuthors.filter(_.resourceId == lift(resourceId)).delete
transaction(run(expr) <&> run(deleteAuthorsAssocs)) // (3)
.mapError:
case e: SQLException => RepositoryError(e)
.unit
end removeResource
This showcases perfectly a situation where we want to execute two queries ((1) & (2)), but only commit them if they both are successful at the same time. This is done by using Quill's transaction, which receives an effect that in this case is the zipping of the two deletes, concurrently. Notice how this time the action for each expression is delete.
Now, for retrieving all the authors associated with a resource, we would need a Join. Quill makes this fairly easy by using the join operator. There are multiple kinds of joins, but on this occasion, we need a simple Left Join.
override def getResourceAuthors(resourceId: Resource.Id): Stream[RepositoryError, Author] =
val expr = quote:
for
ra <- resourceAuthors.filter(ra => ra.resourceId == lift(resourceId))
author <- authors.join(a => ra.authorId == a.id)
yield author
stream(expr).refineOrDie:
case e: SQLException => RepositoryError(e)
end getResourceAuthors
First, we filter those rows at resourceAuthors that have our specified id. Then we join those rows with their respective authors by using query#join.
With this, we have completed our ResourceRepo implementation. As I mentioned the rest are simpler than this one, so it'd be a good exercise to implement those by yourself with the tools you learned.
The material store is where all the files associated with resources will end up. The goal is to let the user do something like:
btek material add 8u4xhdIf --name 'My PDF' download.pdf
btek material add 8u4xhdIf video.mp4
btek material list
# Programming
# [8u4xhdIf] Functional Programming in Scala
# - [Yif39s3f] My PDF.pdf
# - [123dfsl9] video.mp4
To do this, we want a repository that deals with files and a directory, instead of one that deals
with a database. That'll be the job of our MaterialStore.
First, as a dependency of our implementation, we'd need a path that points to the place the user wants to store their files. If you remember in our config section, we set our configuration to have a directory to save our database. Now we want to use the same dir with a 'material' directory appended.
import zio.nio.file.Path
class MaterialStoreLive(storeDir: Path) extends MaterialStore
object MaterialStoreLive:
val layer = ZLayer:
for
cfg <- ZIO.service[AppConfig]
materialDir = cfg.configDir / "material"
_ <- ZIO.attemptBlocking(materialDir.toFile.mkdir())
yield MaterialStoreLive(materialDir)
Now, let's write a method to store a file in our directory.
ZIO NIO offers nice interfaces to use Input/Output operations using, well, ZIO.
libraryDependencies += "dev.zio" %% "zio-nio" % zioVersion
From it we're going to use zio.nio.file.Files, which gives us a lot of tools to work with files (duh).
override def store(
materialId: Material.Id,
name: Material.Name,
file: Path,
): IO[IOError, MaterialSource] =
val destFilename = s"${materialId.value}.${name.value}" // (1)
val destPath = storeDir / destFilename // (2)
val materialSource = MaterialSource(materialId, destPath)
Files
.copy(file, destPath, StandardCopyOption.REPLACE_EXISTING)
.as(materialSource)
.mapError:
case e: IOException => IOError(e)
end store
The destination will be our storeDir / destinationFilename (2), where destFilename is simply the unique identifier of a material, followed by a dot, followed by the material's name (1). Then, by using Files, we're going to copy the path the user passed to us as input to the destPath (2).
We need a method for walking the materials' directory and reading each of the files in there. zio-nio's Files offers a function named walk for exactly that. It receives a path as its parameter and returns a stream of paths. The first of these paths is the directory path we're walking, so we're going to drop it.
override def listMaterialsFiles: Stream[DomainError, MaterialSource] =
Files
.walk(storeDir)
.drop(1)
.map: filePath =>
Material
.Id(filePath.filename.toString.split('.').head)
.map: id =>
MaterialSource(id, filePath) // (1)
.collect:
case Right(material) => material // (2)
.mapError:
case e: IOException => IOError(e)
Remember that we're storing each material with a unique identifier. When traversing the material directory, we have to parse back this id into a MaterialId (1). Since the result of this is an Either that is guaranteed to work for the files we saved ourselves, but not other kinds of files (e.g. those annoying .DS_Store files MacOS autogenerate), we have to collect those right values only (2).
For deleting a material, we first need a helper method for getting one from an id. This method will use our list function to get the stream of files, and then we'd just have to filter and get the head.
private def getMaterialFile(id: Material.Id): IO[DomainError, MaterialSource] =
listMaterialsFiles
.find(_.id == id)
.runHead
.someOrFail(MaterialSourceNotFoundError(id))
Delete a material consists only of getting the material source from an id (with our helper method), pattern matching the source (which is a zio.nio.file.Path), and then using Files.delete with it.
private def deleteMaterial: MaterialSource => IO[DomainError, Unit] =
case MaterialSource(_, source) =>
Files.delete(source).mapError { case e: IOException =>
IOError(e)
}
override def delete(id: Material.Id): IO[DomainError, Unit] =
getMaterialFile(id).flatMap(deleteMaterial)This last layer is described by Gabriel Volpe in his Practical Functional Programming with Scala as a program. A program uses the infrastructure of our application to model actual use cases. These use cases will map 1:1 the commands we established the user must be able to use.
The librarian itself will be just a simple class that receives all our infrastructure on its primary constructor:
class LibrarianLive(
materialStore: MaterialStore,
materialsRepo: MaterialsRepo,
resourcesRepo: ResourcesRepo,
authorsRepo: AuthorsRepo,
categoriesRepo: CategoriesRepo,
):
// ...For a user to create a resource, we have to create the authors and the category associated with it if it doesn't exist yet. If a category or an author is not found, we're making the effect fail with a NotFoundError, so if we catch one of them while fetching the category or author, then we'd need to create a new one from there.
private def createCategoryIfDoesNotExist(name: Category.Name): IO[DomainError, Category] =
categoriesRepo
.getByName(name)
.catchSome:
case NotFoundError(_) =>
categoriesRepo.insert(Category(name)).as(Category(name))
private def createAuthorIfDoesNotExist(name: String): IO[DomainError, Author] =
authorsRepo
.getByName(name)
.catchSome:
case NotFoundError(_) =>
for
id <- Author.Id.make
author = Author(id, name)
_ <- authorsRepo.insert(author)
yield author
With these helpers, let's add our function for creating resources. We receive authors as a set and a category as an optional.
/** Creates a resource and its associated authors and categories if they don't not exist, fetching
* them by name
*/
def createResource(
title: String,
kind: ResourceKind,
category: Option[Category.Name],
publicationYear: Option[Short],
authors: Set[String],
): IO[DomainError, ResourceRow] =
val resourceData = ResourceData(title, kind, publicationYear, category)
for
authors <- ZIO.foreach(authors)(createAuthorIfDoesNotExist(_)) // (1)
_ <- ZIO.fromOption(category).foldZIO(_ => ZIO.none, createCategoryIfDoesNotExist(_).asSome) // (2)
resource <- resourcesRepo.addResource(resourceData) // (3)
_ <- resourcesRepo.addAuthorsToResource(resource.id, authors.map(_.id).toSeq*) // (4)
yield resource
end createResource
ZIO.foreach, which takes up a list and applies a ZIO to each one, returning a sequence wrapped in a ZIO effect.resourceRepo to create a new resource.For a resource to be displayed with all its relevant information, we have to fetch not only the row as it is from the database but also retrieve what's associated with it, namely a category and its authors.
private def resourceFromRow(row: ResourceRow): IO[DomainError, Resource] =
for
category <-
ZIO.fromOption(row.categoryName).foldZIO(_ => ZIO.none, categoriesRepo.getByName(_).asSome) // (1)
authors <- resourcesRepo.getResourceAuthors(row.id).runCollect.map(_.toSet) // (2)
yield Resource(row.id, row.title, row.kind, category, row.publicationYear, authors)
Here we use the same pattern as before, where we build a ZIO effect from an option and fold over it (1), providing a none as default and a some if we get a category name to work with. Authors related to a resource are fetched as a stream (2), so we need to runCollect all of them. This gives us a chunk so we have to turn it into a set first (2).
private def listResourcesFromRows(categoryOpt: Option[Category.Name] = None) =
resourcesRepo.listResources(categoryOpt).mapZIO(resourceFromRow(_))
With this helper that gives us a stream of Resources, the last thing we have to do is to group resources by their category, so they can be displayed contiguously. For this reason and because we need to do the same thing when displaying materials, I created a helper method for streams that uses groupByKey, and just returns them aside the key we're grouping with:
import zio.stream.*
import zio.*
extension [R, E, O](s: ZStream[R, E, O])
def simpleGroupByKey[K](f: O => K): ZStream[R, E, (K, O)] = s.groupByKey(f): (k, ss) =>
ss.map(k -> _)
Having a stream of grouped tuples, we can create a map from their key to a stream of those grouped records. We'll have another extension method for that:
extension [R, E, K, O](s: ZStream[R, E, (K, O)])
def runCollectMap: ZIO[R, E, Map[K, UStream[O]]] =
s.runFold(Map.empty[K, UStream[O]]):
case (s, (k, t)) =>
s + (k -> (s.get(k).getOrElse(ZStream.empty) ++ ZStream.succeed(t)))
With this implementation, we'd have to load all the records in memory, which is not ideal, but gives us the ability to work with them still using streams, which is good for when we build a string of outputs for each resource record.
Now let's pull everything together:
/** @return Resources grouped by category name, `"Root"` being the default. */
def listResources(
categoryOpt: Option[Category.Name],
): IO[DomainError, Map[Category.Name, UStream[Resource]]] =
listResourcesFromRows(categoryOpt)
.simpleGroupByKey(_.category.getOrElse(Category.root).name)
.runCollectMap
For resources without a category we're setting Category.root as fallback.
Deleting a resource is quite simple. We're just going to use our repository method for that:
def deleteResource(resourceId: Resource.Id): IO[DomainError, Unit] =
resourcesRepo.removeResource(resourceId)To add material, we first need to create the material in the database because we need its id to actually store the file in our directory.
def addMaterial(
name: Option[String],
resourceId: Resource.Id,
file: JPath,
): IO[DomainError, Material] =
val filePath = Path.fromJava(file)
for
material <- materialsRepo.addMaterial(MaterialData(name, resourceId, filePath)) // (1)
_ <- materialStore
.store(material.id, material.name, filePath) // (2)
.tapError: _ =>
materialsRepo.deleteMaterial(material.id) // (3)
yield material
end addMaterial
We create the material using the MaterialData case class to group the data (1). Then, using the newly created id, we can store the file we're receiving as a function argument (2). This time, however, we want to make sure no material exists only in the database when, for example, storing fails (which can easily be the case). For this reason, we want to delete the Database material if something goes wrong with the storing function (3). ZIO's tapError makes this quite easy.
All we needed for this one we've already done it. We need to group records by key, the category, and then collect them grouped as a map. The only thing that changes is that for each resource, we need to fetch their materials from the database (1).
def listMaterials(
resourceIdOpt: Option[Resource.Id],
): IO[DomainError, Map[Category.Name, UStream[(Resource, List[Material])]]] =
listResourcesFromRows()
.mapZIO(r => materialsRepo.listMaterials(Some(r.id)) /* (1) */.runCollect.map(r -> _.toList))
.simpleGroupByKey(_._1.category.getOrElse(Category.root).name)
.runCollectMap
end listMaterialsIf you remember, we want to add certain flexibility at the moment of removing materials. We want the user to be able to type something like:
btek material list
# Programming
# [8u4xhdIf] Functional Programming in Scala
# - [Yif39s3f] My PDF.pdf
# - [123dfsl9] video.mp4
# Deletes only the specified material.
btek material remove Yif39s3f
# Removes the specified material under
btek material remove 8u4xhdIf video.mp4
# Removes all material associated with the resource.
btek material remove 8u4xhdIf
These use cases translate to:
Material.Id to remove a specific material.Resource.Id and a material name to remove a material, by name, associated with the given resource.Resource.Id, without a name.Let's say we have the material we're going to remove:
def deleteFromStoreWithRevert(material: Material) = materialStore
.delete(material.id)
.tapError: _ =>
materialsRepo.addMaterial(material) // (1)
If the deletion from the store fails, we'd like to restore the material into the database (1). With this helper, if the user passes a Material.Id, then have to try to:
val fromMaterialId =
for
materialId <-
ZIO.fromEither(Material.Id(resourceOrMaterialId)).mapError(e => NewtypeBuildError(e)) // (1)
material <- materialsRepo.getMaterialById(materialId) // (2)
_ <- materialsRepo.deleteMaterial(materialId) // (3)
_ <- deleteFromStoreWithRevert(material) // (4)
yield List(materialId)
In case the user passes a Resource.Id, things get more complicated.
val fromResourceId =
for
resourceId <-
ZIO.fromEither(Resource.Id(resourceOrMaterialId)).mapError(e => NewtypeBuildError(e)) // (1)
materials <-
materialsRepo
.listMaterials(Some(resourceId)) // (2)
.filter(m => materialNameOpt.fold(true)(materialName => m.name == materialName)) // (3)
.runCollect
.map(_.toList)
.flatMap: // (4)
case Nil =>
ZIO.fail:
NotFoundError(
s"$resourceOrMaterialId${materialNameOpt.fold("")(name => s" $name")}",
)
case list => ZIO.succeed(list)
_ <- ZIO.foreachDiscard(materials)(material => materialsRepo.deleteMaterial(material.id)) // (5)
_ <- ZIO.foreachDiscard(materials)(material => deleteFromStoreWithRevert(material)) // (6)
yield materials.map(_.id)
Resource.Id. This might fail.true will do. Otherwise, we compare them.foreachDiscard executes a ZIO for each value, but discards their results. We just want to know if it succeeds or it fails.Materials repo offers to pass an optional Resource.Id as a filter to get materials. Is defined as follows: override def listMaterials(
resourceIdOpt: Option[Resource.Id] = None,
): Stream[DomainError, Material] =
val expr = resourceIdOpt.fold(quote(materials)): ri =>
quote:
materials.filter(_.resourceId == lift(ri))
stream(expr).refineOrDie(RepositoryError(_))
end listMaterials
I hope this was bearable. With all the interfaces we had to work with we abstracted away the implementation details of each, and focused on composing those into a cohesive program, and that allowed us to keep busy thinking about the user experience and not much about the details of datastores and more. Notice how we choose where and how to deal with errors thrown in the previous layer, or if we take care of them at all in this layer. This is one of the greatest benefits of using effect systems. Our whole program is wrapped with its context and we choose how much of that context is important for us at any given time.
With all this, we're almost there. The only piece missing so far is the presentation layer, where our users will trigger our program and pass arguments to it.
For our CLI interface, we're going to use the official ZIO library zio-cli. Although I'd say it lacks a couple of features like parsing multiple option flags or optional arguments, I have to say that using this library was a breeze to use.
To create the structure of nested subcommands we'll use Algebraic Data Types. Let's go through each command and subcommand bit by bit.
We're going to have two main sub-commands, namely resources and materials. For this reason, it makes sense to have a Subcommand ADT with two subclasses.
sealed trait Subcommand extends Product with Serializable
object Subcommand:
sealed trait Resources extends Subcommand
sealed trait Materials extends Subcommand
Both resources and materials have subcommands, so let's create an ADT for each one.
// ...
object Subcommand:
sealed trait Resources extends Subcommand
object Resources:
sealed trait ResourcesSubcommand extends Subcommand
sealed trait Materials extends Subcommand
object Materials:
sealed trait MaterialsSubcommand extends Subcommand
create The first command in our pile is create. See how it works:
# create a resource
btek resources create --kind book --title "Functional Programming in Scala" --authors "Michael Pilquist, Runar Bjarnason, Paul Chiusano" --category Programming --published 2024
First let's get our ADT for the subcommand.
// ...
object Subcommand:
// ...
object Resources:
// ...
object ResourcesSubcommand:
final case class Create(
title: String,
kind: ResourceKind,
category: Option[Category.Name],
publicationYear: Option[Short],
authors: Set[String],
) extends ResourcesSubcommand
// ...
zio-cli comes with its ZIOAppDefault, ZIOCliDefault. This App wrapper will give us a function to implement: cliApp. This function receives a command builder, and after it's built it'll give us the resulting subcommand case class, which we can pattern match to feed their values into our actual program.
override def cliApp = CliApp
.make("btek", "0.1.0-SNAPSHOT", HelpDoc.Span.text("sample docs"), btek):
case Subcommand.Resources.ResourcesSubcommand.List => ???
But how do we describe how we want to build those case classes?
For this purpose, ZIO CLI has two kinds of descriptors: Args & Options. They describe, as it is evident, arguments and options, respectively, a command needs. They both have static constructors that describe a type to be captured by them, such as text, localDate or even directory. They both might be combined with others in kind of the same way: with the ++ operator.
Behind the scenes, this operator gets translated into a tuple of arguments and options that can be mapped into our case classes.
import zio.cli.*
val resourcesCreate = Command(
"create",
Options.text("title") ++
Options.text("kind").mapTry(s => ResourceKind.valueOf(s.toLowerCase.capitalize))
++ Options.text("category").map(Category.Name(_)).optional
++ Options.integer("publication-year").map(_.toShort).optional
++ Options.text("authors").??("author list separated by commas").alias("author").optional,
).map:
case (title, kind, category, publicationYear, authorsOpt) =>
Subcommand.Resources.ResourcesSubcommand.Create(
title,
kind,
category,
publicationYear,
authorsOpt.map(authors => authors.split(",").map(_.trim()).toSet).getOrElse(Set()),
)
Now, when we create the CliApp, we can pattern match the result of the arguments the user passed, converted into our case classes.
// ...
case ResourcesSubcommand.Create(t, k, c, p, a) =>
librarian
.createResource(t, k, c, p, a)
.fold(
e => s"An error has occurred while creating the resource: ${e.message}",
r => s"Resource with id ${r.id} was created",
)
.flatMap(zio.Console.printLine(_))
listFrom now on we're going to move fast since we've already got the basis for building CLI commands and using them. To print a nice list of resources, we gotta create a helper function that receives the Map[Category.Name, UStream[Resource]] we build in the librarian chapter, using our stream helper functions.
Each resource is going to be shown with its authors and year of publication, nicely displaying and handling the cases where authors or a publication year is not available:
private def showResource(r: Resource): String =
def showMeta: Resource => String =
r =>
val authors = r.authors.map(_.name).mkString(", ")
val publicationYear = r.publicationYear.map(_.toString).getOrElse("")
if !authors.isBlank() || !publicationYear.isBlank() then
s"(${List(authors, publicationYear).filterNot(_.isBlank()).mkString(", ")})"
else ""
val meta = showMeta(r)
List(s"[${r.id}]", r.title, meta).filterNot(_.isBlank()).mkString(" ") ++ "."
end showResource
Each category is displayed as a header, followed by the resources associated with it, with a bit of indentation.
private def showCategory(
categoryName: Category.Name,
resources: UStream[(Resource, List[Material])],
): UStream[String] =
ZStream(categoryName.value.capitalize) ++ resources.flatMap: (r, ms) =>
ZStream("\t- " ++ showResource(r)) ++ ZStream
.fromIterable(ms)
.map(m => "\t\t* " ++ showMaterial(m))
For now, let's ignore the fact of displaying material associated with a resource. When we're displaying only resources, that's the same as displaying them with an empty material list associated.
def showResources(resourcesMap: Map[Category.Name, UStream[Resource]]): UStream[String] =
showResourcesWithMaterials(resourcesMap.map((k, v) => (k, v.map(_ -> List.empty))))
This is how we show resources with materials:
def showResourcesWithMaterials(
resourcesMap: Map[Category.Name, UStream[(Resource, List[Material])]],
): UStream[String] =
val rootOpt = resourcesMap.get(Category.root.name)
val noRoot = resourcesMap - Category.root.name
val rootShow =
rootOpt.fold(ZStream.empty)(rootResources => showCategory(Category.root.name, rootResources))
rootShow ++ ZStream.fromIterable(noRoot).flatMap(showCategory)
end showResourcesWithMaterials
The command building looks like this. It receives at most 1 category name (that means it can also receive 0 arguments), so when we build the `Option[Category.Name]`, we'd have to make sure to take only the head as an option.
Command ADT:
object Subcommand:
// ...
object Resources:
// ...
object ResourcesSubcommand:
// ...
final case class List(category: Option[Category.Name] = None) extends ResourcesSubcommand
// ...
val resourcesList =
Command("list", Args.text("category").map(Category.Name(_)).atMost(1))
.map(category => Subcommand.Resources.ResourcesSubcommand.List(category.headOption))
.withHelp("Lists all resources or under a single ")
We already have what we need to process this command, our Librarian list method.
case ResourcesSubcommand.List(category) =>
librarian
.listResources(category)
.map(CliDisplayer.showResources(_))
.flatMap(s => s.runForeach(zio.Console.printLine(_)))
Here we use the showResources method we created previously. Remember that it yields a stream of strings, each corresponding to a line, which we print to the console.
removeTo remove resources, we expose the remove command which receives a resource id, which we can map right inside the command so zio-cli can report the error to the user if the id is typed in the wrong format. We map the left value from a BuildFailure which is the type of monix.newtype failures, to a HelpDoc, which is a sort of DSL for building documentation that can be displayed with some format by ZIO CLI.
Command ADT:
// ...
object Subcommand:
// ...
object Resources:
// ...
object ResourcesSubcommand:
// ...
final case class Remove(id: Resource.Id) extends ResourcesSubcommand
val resourcesRemove =
Command(
"remove",
Args
.text("resource-id")
.mapOrFail(Resource.Id(_).left.map(e => HelpDoc.p(e.toReadableString))),
).map(id => Subcommand.Resources.ResourcesSubcommand.Remove(id))
Processing this command requires us to just call the delete method from our Librarian.
// ...
case ResourcesSubcommand.Remove(resourceId) =>
librarian
.deleteResource(resourceId)
.map(_ => s"Resource $resourceId was successfully deleted.")
.flatMap(zio.Console.printLine(_))
With that, we're completely set to manage resources from our CLI! There's one last piece missing, Materials.
Materials are the main reason for this app to exist as a CLI. It makes it easy to manage files associated with a resource we've been watching or reading.
addAdd has an interesting argument from the CLI point of view. It receives a file path as its last argument. This can be given in absolute or relative terms just as with any other CLI program we use. We can instruct ZIO CLI to make sure this file actually exists or fails otherwise by passing a zio.cli.Exists argument to the Args.file descriptor. Exists is simply a boolean whose purpose, I suppose, is to make very very clear your intentions.
Command ADT:
object Subcommand:
// ...
object Materials:
// ...
object MaterialsSubcommand:
// ...
final case class Add(name: Option[String], resourceId: Resource.Id, file: JPath) extends MaterialsSubcommand
val materialsAdd =
Command(
"add",
Options.text("name").optional,
Args
.text("resource-id")
.mapOrFail(Resource.Id(_).left.map(e => HelpDoc.p(e.toReadableString)))
++ Args.file("file", Exists.Yes),
).map:
case (name, (resourceId, file)) =>
Subcommand.Materials.MaterialsSubcommand.Add(name, resourceId, file)
Apart from that, this has nothing you haven't already seen in this tutorial. The actionable part is also quite boring (a good thing while programming).
// ...
case MaterialsSubcommand.Add(name, resourceId, file) =>
librarian.addMaterial(name, resourceId, file)
listWe're going to fly over materials list since we already prepared the ground while listing resources. Remember we created a couple of helper functions to group and then collect streams into a map from a key? We'll do that again, but this time, collecting the material associated with each resource. The command itself receives an optional Resource.Id, which serves to filter and show material only for a single resource.
Command ADT:
// ...
object Subcommand:
// ...
object Materials:
// ...
object MaterialsSubcommand:
// ...
final case class List(resourceId: Option[Resource.Id]) extends MaterialsSubcommand
val materialsList = Command(
"list",
Args
.text("resource-id")
.mapOrFail(Resource.Id(_).left.map(e => HelpDoc.p(e.toReadableString)))
.atMost(1),
).map: resourceId =>
Subcommand.Materials.MaterialsSubcommand.List(resourceId.headOption)
When receiving a list command, we process it like this:
case MaterialsSubcommand.List(resourceId) =>
librarian
.listMaterials(resourceId)
.flatMap: resourcesMap =>
CliDisplayer
.showResourcesWithMaterials(resourcesMap)
.runForeach(zio.Console.printLine(_))
We use the same displayer as before: showResourcesWithMaterials. This 'displayer' has the peculiarity of displaying, surprise, materials.
private def showMaterial(m: Material): String =
s"[${m.id}] ${m.name}"
removeRemove is our last use-case. Yay!
Here, we're not going to parse the id received from the CLI directly because we don't know if it is either a Resource.Id or a Material.Id. We also receive an optional material name which will only be used in case we get a Resource.Id as the first argument (because getting a material id means we are certain which exact material to delete). Again, we model optional arguments by using the atMost(1), because ZIO CLI doesn't support it any other way.
Command ADT:
object Subcommand:
// ...
object Materials:
// ...
object MaterialsSubcommand:
// ...
final case class Remove(resourceOrMaterialId: String, name: Option[Material.Name])
extends MaterialsSubcommand
def materialsRemove = Command(
"remove",
Args
.text("resource-or-material-id") ++ Args.text("material-name").map(Material.Name(_)).atMost(1),
).map: (resourceOrMaterialId, materialName) =>
Subcommand.Materials.MaterialsSubcommand.Remove(resourceOrMaterialId, materialName.headOption)
Grouping all material subcommands needs to use the same method as with resources, subcommands:
val materials = Command("materials")
.withHelp("help")
.subcommands(materialsAdd, materialsList, materialsRemove)
The program when removing goes as usual, with reporting of each material that was deleted:
case MaterialsSubcommand.Remove(resourceOrMaterialId, materialName) =>
librarian
.removeMaterial(resourceOrMaterialId, materialName)
.map: mis =>
"The following materials were deleted:\n" ++ mis
.map(mi => s"\t* Material ${mi.value}\n")
.mkString
.flatMap(zio.Console.print(_))
btekWe gotta glue all the commands we created and map into a sealed tree by using the subcommands method we used for materials and resources.
val btek = Command("btek", Options.none, Args.none).subcommands(resources, materials)
As shown previously, this makes all commands fit into the tree-like subcommand structure, which then we could map into the little programs we've already shown.
override def cliApp = CliApp
.make("btek", "0.1.0-SNAPSHOT", HelpDoc.Span.text("sample docs"), btek): cmd =>
val program =
ZIO
.service[LibrarianLive]
.flatMap: librarian =>
cmd match
case // Subcommands =>
I added another command: setup. This command will run the setupDatabase function since is quite resource-consuming and we want to avoid running it every time the user uses the application. The implementation has nothing of interest other than matching the command and running the function, so I won't show it here.
Our val program = ... needs every dependency in place to work. To get the service LibrarianLive we need a librarian layer. This layer needs all of our infrastructure layer, which compresses our repositories and our store. These layers, in turn, need the app layer, which contains our data source and configuration layers.
This sort of tree-like structure can be expressed simply by writing:
program.provide(
AppConfig.layer,
dataSourceLayer,
sqliteLayer,
MaterialStoreLive.layer,
MaterialsRepoLive.layer,
ResourcesRepoLive.layer,
AuthorsRepoLive.layer,
CategoriesRepoLive.layer,
LibrarianLive.live,
)
I like an approach that makes clear how each layer depends upon a previous one, resembling the tree structure that actually is. For that, layers support syntax to express this kind of dependency tree. This is what I came up with for this example:
val appLayer = AppConfig.layer >+> dataSourceLayer >+> sqliteLayer
val infraLayer = appLayer >>>
(MaterialStoreLive.layer ++
MaterialsRepoLive.layer ++
ResourcesRepoLive.layer ++
AuthorsRepoLive.layer ++
CategoriesRepoLive.layer)
val programLayer = infraLayer >>> (LibrarianLive.live ++ appLayer)
I hope some of my limited knowledge has been successfully passed to you through this article. My intention was to showcase how some ZIO features make it so easy to approach software development with layers, streams, fallbacks and retries, by coding something I found fun to program. The way I learn best is by detailed explanations, using interesting to keep my attention, of a limited set of features, because once I get in the vibe I can pick up the rest by myself, and that's what tried to do with this article.
We used, rather thoroughly, several key libraries in the ecosystem, namely ZIO's Quill, and in case you see value in creating CLIs, which I do since I've already shipped one for production usage, ZIO CLI.
This program, although complete and, let's say, 'useful', is not without its shortcomings and deficiencies. I highly encourage the reader to see this as an opportunity to extend and improve the code that today was presented. Here are a few ideas:
Stay ahead with the latest insights and breakthroughs from the world of technology. Our newsletter delivers curated news, expert analysis, and exclusive updates right to your inbox. Join our community today and never miss out on what's next in tech.