Skip to content

Commit

Permalink
user manual and readme
Browse files Browse the repository at this point in the history
  • Loading branch information
@balazskreith
balazskreith committed Aug 8, 2024
1 parent d6e25ce commit aefb702
Show file tree
Hide file tree
Showing 9 changed files with 632 additions and 518 deletions.
102 changes: 65 additions & 37 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,35 @@
Hamok
---
Hamok is a library provides distributed object storages.
It is developed on top of [Raft](https://raft.github.io/) consensus algorithm.
Here's the combined README for the Hamok library:

# Hamok Library

Hamok is a distributed object storage library developed using the [Raft](https://raft.github.io/) consensus algorithm. It provides a framework for building reliable and fault-tolerant distributed systems by enabling synchronized data operations across multiple nodes in a network.

## Installation

To use the Hamok library in your project, install it via npm:

```bash
npm install hamok
```

Or via yarn:

```bash
yarn add hamok
```

## Table of Contents

- [Quick Start](#quick-start)
- [Concept](#concept)
- [Collections](#collections)
- [HamokMap](#hamokmap)
- [HamokQueue](#hamokqueue)
- [HamokEmitter](#hamokemitter)
- [HamokRecord](#hamokrecord)
- [User Manual](#user-manual)
- [Contributing](#contributing)
- [License](#license)
## Quick Start

```javascript
Expand Down Expand Up @@ -53,58 +80,59 @@ import { Hamok } from 'hamok';
})();
```

## Table of Contents
* [Concept](#concepts)
* [Storages and Collections](#storages)
* [Storage Grids](#storage-grids)
*

## Concept

Hamok is a lightweight, distributed object storage library developed using [Raft](https://raft.github.io/) consensus
algorithm. Hamok provides distributed map, queue, event emitters, and record object.
It is designed to minimize setup effort and maximize flexibility, it offers the essential
logic to embed its library and utilize shared storage, enabling efficient object sharing across service instances.
Hamok is a lightweight, distributed object storage library developed using the [Raft](https://raft.github.io/) consensus algorithm. Hamok provides distributed map, queue, event emitters, and record object. It is designed to minimize setup effort and maximize flexibility, offering the essential logic to embed its library and utilize shared storage, enabling efficient object sharing across service instances.

### Hamok on RAFT

[Raft](https://raft.github.io/) is a consensus algorithm designed to manage a replicated log across a distributed system.
Its primary goal is to ensure that multiple servers agree on a sequence of state transitions, providing consistency
and fault tolerance in distributed systems. RAFT breaks down the consensus problem into three subproblems:
[Raft](https://raft.github.io/) is a consensus algorithm designed to manage a replicated log across a distributed system. Its primary goal is to ensure that multiple servers agree on a sequence of state transitions, providing consistency and fault tolerance in distributed systems. RAFT breaks down the consensus problem into three subproblems:

- **Leader Election**: Ensures that one server acts as the leader, which is responsible for managing the log replication.
- **Leader Election:** Ensures that one server acts as the leader, which is responsible for managing the log replication.

- **Log Replication**: The leader receives log entries from clients and replicates them to follower servers. The leader waits for a majority of followers to acknowledge the entries before considering them committed.
- **Log Replication:** The leader receives log entries from clients and replicates them to follower servers. The leader waits for a majority of followers to acknowledge the entries before considering them committed.

- **Safety**: RAFT guarantees that committed log entries are durable and will not be lost, even in the presence of server failures. It ensures that no two leaders can be elected for the same term and that logs are consistent across servers.
- **Safety:** RAFT guarantees that committed log entries are durable and will not be lost, even in the presence of server failures. It ensures that no two leaders can be elected for the same term and that logs are consistent across servers.

Overall, RAFT is designed to be understandable and easy to implement while providing strong consistency and reliability in distributed systems.

Hamok uses Raft to manage the shared storage accross multiple instances.
Hamok uses Raft to manage the shared storage across multiple instances.

### Features

- **Raft-based Consensus:** Ensures consistent data replication across nodes.
- **Distributed Data Structures:** Provides maps, queues, records, and emitters.
- **Event-driven Architecture:** Emits events for state changes, errors, and communication.


## Collections

### HamokMap

HamokMap is a distributed map implementation that leverages the RAFT algorithm to ensure consistency and fault tolerance. It provides a key-value store that can be accessed and modified by multiple service instances, allowing for efficient data sharing and synchronization across the system.

### HamokQueue

HamokQueue is a distributed queue that allows for asynchronous FIFO-type message passing between service instances. Using RAFT, it maintains the order and durability of messages, ensuring that all instances have a consistent view of the queue contents and can process messages reliably.

### HamokEmitter

## HamokMap
HamokEmitter is an event emitter designed for distributed systems. It allows service instances to emit and listen to events, facilitating communication between instances.

HamokMap is a distributed map implementation that leverages the RAFT algorithm to ensure
consistency and fault tolerance. It provides a key-value store that can be accessed and
modified by multiple service instances, allowing for efficient data sharing and
synchronization across the system.
### HamokRecord

## HamokQueue
HamokRecord is a feature that provides distributed storage for individual record objects. Each record can be accessed and updated by multiple service instances, with RAFT ensuring that all updates are consistently applied and persisted across the system.

HamokQueue is a distributed queue that allows for asynchronous FIFO type message passing between service instances.
Using RAFT, it maintains the order and durability of messages, ensuring that all instances have a
consistent view of the queue contents and can process messages reliably.

## HamokEmitter
## User Manual

HamokEmitter is an event emitter designed for distributed systems. It allows service instances to emit and listen to events,
facilitating communication between instances.
You can find detailed user manuals [here](www.hamok.dev)

## HamokRecord

HamokRecord is a feature that provides distributed storage for individual record objects.
Each record can be accessed and updated by multiple service instances,
with RAFT ensuring that all updates are consistently applied and persisted across the system.
## Contributing

Contributions are welcome! Please feel free to submit issues or pull requests to improve the library.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
162 changes: 80 additions & 82 deletions docs/emitter.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,33 @@
* [Examples](#examples)
* [FAQ](#faq)

## Table of Contents
* [Overview](#overview)
* [Configuration](#configuration)
* [API Reference](#api-reference)
* [Properties](#properties)
* [Events](#events)
* [Methods](#methods)
* [Examples](#examples)
* [FAQ](#faq)


## Overview

`HamokEmitter` is a class that provides a mechanism for managing distributed event subscriptions and publishing events across multiple nodes in a distributed system. It integrates with `HamokConnection` for communication and uses `EventEmitter` for event handling.

## API Reference `HamokEmitter<T extends HamokEmitterEventMap>`

### Create a HamokEmitter instance

To create a `HamokEmitter` instance, you need a `Hamok` instance. Here is how you can create a `HamokEmitter` instance:

```typescript
const emitter = hamok.createEmitter<MyEventMap>({
emitterId: 'exampleEmitter',
});
```

### Configuration

```typescript
type MyEventMap = {
myEvent: [string, number];
Expand All @@ -35,132 +52,100 @@ const emitter = hamok.createEmitter<MyEventMap>({

/**
* Optional. The timeout duration in milliseconds for requests.
*
* DEFAULT: 5000
*/
requestTimeoutInMs: 5000,

/**
* Optional. The maximum waiting time in milliseconds for a message to be sent.
* The storage holds back the message sending if Hamok is not connected to a grid or not part of a network.
*
* DEFAULT: 10x requestTimeoutInMs
*/
maxMessageWaitingTimeInMs: 50000,

/**
* Optional. The maximum number of keys allowed in request or response messages.
*
* DEFAULT: 0 means infinity
*/
maxOutboundMessageKeys: 1000,

/**
* Optional. The maximum number of values allowed in request or response messages.
*
* DEFAULT: 0 means infinity
*/
maxOutboundMessageValues: 100,

/**
* Optional. A map of payload codecs for encoding and decoding event payloads.
* The key is an event type, and the value is a codec for that event type.
*
* DEFAULT: JSON codec
*/
payloadsCodec?: Map<keyof T, { encode: (...args: unknown[]) => string, decode: (data: string) => unknown[] }>,
payloadsCodec?: Map<keyof MyEventMap, { encode: (...args: unknown[]) => string, decode: (data: string) => unknown[] }>,
});
```

### Configuration
## API Reference

```typescript
const emitter = hamok.createEmitter<MyEventMap>({
emitterId: 'exampleEmitter',
});
```
### `HamokEmitter<T extends HamokEmitterEventMap>` Class

### Events
A class for managing events and subscriptions in a distributed system.

The `HamokEmitter` class emits event given the event keys defined in the `HamokEmitterEventMap`. Each event will pass the arguments defined in the event map to its listeners.
#### Properties

```typescript
await emitter.subscribe('myEvent', (message, count) => {
console.log(`Received: ${message} - ${count}`);
});
```
- `id`: `string` - The unique identifier of the emitter.
- `closed`: `boolean` - Indicates whether the emitter is closed.
- `connection`: `HamokConnection<string, string>` - The connection used by the emitter.
- `payloadsCodec`: `Map<keyof T, { encode: (...args: unknown[]) => string, decode: (data: string) => unknown[] }>` - Optional codec for encoding and decoding payloads.

### Properties
#### Methods

- **id**: `string` - The unique identifier for the `HamokEmitter` instance.
- **closed**: `boolean` - Indicates whether the emitter is closed.
- **close**(): `void` - Closes the emitter and releases any held resources.
- **subscribe**<K extends keyof T>(`event: K`, `listener: (...args: T[K]) => void`): `Promise<void>` - Subscribes a listener to an event.
- **unsubscribe**<K extends keyof T>(`event: K`, `listener: (...args: T[K]) => void`): `Promise<void>` - Unsubscribes a listener from an event.
- **clear**(): `void` - Clears all subscriptions and listeners.
- **publish**<K extends keyof T>(`event: K`, `...args: T[K]`): `Promise<string[]>` - Publishes an event to all subscribed listeners.
- **notify**<K extends keyof T>(`event: K`, `...args: T[K]`): `void` - Notifies all subscribed listeners of an event.
- **export**(): `HamokEmitterSnapshot` - Exports the current state of the emitter.
- **import**(`snapshot: HamokEmitterSnapshot`): `void` - Imports the state from a snapshot.

### Methods
### Events

#### `close()`
- `InsertEntriesRequest`: Manages subscription and adds the source endpoint to the list.
- `RemoveEntriesRequest`: Manages subscription and removes the source endpoint from the list.
- `UpdateEntriesRequest`: Emits events with decoded payloads.
- `UpdateEntriesNotification`: Emits events with decoded payloads.
- `ClearEntriesNotification`: Manages subscription and removes the source endpoint from the list.
- `remote-peer-removed`: Removes the remote peer from all subscriptions.
- `close`: Closes the emitter and removes all listeners.

Closes the `HamokEmitter` instance, stopping all event subscriptions and communication.
### Example Usage

```typescript
emitter.close();
```

#### `subscribe<K extends keyof T>(event: K, listener: (...args: T[K]) => void): Promise<void>`

Subscribes a listener to an event.
const emitter = new HamokEmitter(connection, payloadsCodec);

```typescript
await emitter.subscribe('myEvent', (message, count) => {
console.log(`Received: ${message} - ${count}`);
emitter.subscribe('event', (data) => {
console.log(`Received data: ${data}`);
});
```

#### `unsubscribe<K extends keyof T>(event: K, listener: (...args: T[K]) => void): Promise<void>`

Unsubscribes a listener from an event.

```typescript
await emitter.unsubscribe('myEvent', (message, count) => {
console.log(`Received: ${message} - ${count}`);
emitter.publish('event', 'sample data').then((peerIds) => {
console.log(`Event published to peers: ${peerIds}`);
});
```

#### `clear()`

Clears all event subscriptions.

```typescript
emitter.clear();
```

#### `publish<K extends keyof T>(event: K, ...args: T[K]): void`

Publishes an event to all subscribed listeners.
It returns a promise that resolves when the event is published on all remote peers subscribed to this event.

```typescript
await emitter.publish('myEvent', 'Hello, world!', 42);
```

#### `notify<K extends keyof T>(event: K, ...args: T[K]): void`

Notifies all subscribed listeners of an event without waiting for the event to be published on remote peers.

```typescript
emitter.notify('myEvent', 'Hello, world!', 42);
```


#### `export(): HamokEmitterSnapshot`

Exports the current state of the emitter, including all event subscriptions. (Used by the `Hamok` class to export the entire state of the Hamok instance.)

```typescript
const snapshot = emitter.export();
console.log(snapshot);
```

#### `import(snapshot: HamokEmitterSnapshot): void`

Imports a previously exported emitter state. (Used by the `Hamok` class to import the entire state of the Hamok instance.)
emitter.unsubscribe('event', (data) => {
console.log(`Unsubscribed from event`);
});

```typescript
emitter.import(snapshot);
emitter.close();
```

## Examples

- [simple distributed emitter](../examples/src/emitter-example.ts)
- [simple distributed emitter](https://github.com/balazskreith/hamok-ts/blob/main/examples/src/emitter-example.ts)

## FAQ

Expand Down Expand Up @@ -239,3 +224,16 @@ To publish an event, use the `publish` method:
```typescript
emitter.publish('myEvent', 'Hello, world!', 42);
```

### What is the difference between the `publish` and `notify` methods?

The `publish` method publishes an event to all subscribed listeners and waits for the event to be published on remote peers.
then return a promise that resolves when the event is published on all remote peers subscribed to this event.

The `notify` method notifies all subscribed listeners of an event without waiting for the event to be published on remote peers.

In short use notify for fire and forget messages, and use publish for messages that need to be checked if it is delivered to the target remote peers.

### What is the payloadsCodec for?

The `payloadsCodec` is a map of payload codecs for encoding and decoding event payloads. The key is an event type, and the value is a codec for that event type. This is useful for customizing the encoding and decoding of event payloads, if you are for example unsatisfied with the default JSON encoding/decoding.
Loading

0 comments on commit aefb702

Please sign in to comment.