Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Windows support, including openssh and pageant agents; closes #19 #35

Merged
merged 4 commits into from
Dec 20, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
/lorg
/.last-testcase
/orgalorg
/orgalorg.exe
/.cover
/coverage
1 change: 1 addition & 0 deletions .goreleaser.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ builds:
goos:
- linux
- darwin
- windows
goarch:
- amd64
flags:
Expand Down
20 changes: 9 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@
fail on any error or try to complete, no matter of what.

* Interactive password authentication as well as SSH public key authentication.
Will use ssh-agent if present. On Windows, orgalorg can connect to
**pageant** or **openssh agent**.

* Ability to run commands through `sudo`.

Expand All @@ -47,18 +49,18 @@ go get github.com/reconquest/orgalorg

# Alternatives

* ansible: intended to apply complex DSL-based scenarios of actions;
* ansible: intended to apply complex DSL-based scenarios of actions;
orgalorg aimed only on running commands and synchronizing files in parallel.
orgalorg can accept target hosts list on stdin and can provide realtime
output from commands, which ansible can't do (like running `tail -f`).
orgalorg also uses same argument semantic as `ssh`:
orgalorg also uses same argument semantic as `ssh`:
`orgalorg ... -C tail -f '/var/log/*.log'` will do exactly the same.

* clusterssh / cssh: will open number of xterm terminals to all nodes.
* clusterssh / cssh: will open number of xterm terminals to all nodes.
orgalorg intended to use in batch mode, no GUI is assumed. orgalorg, however,
can be used in interactive mode (see example section below).

* pssh: buggy, uses binary ssh, which is not resource efficient.
* pssh: buggy, uses binary ssh, which is not resource efficient.
orgalorg uses native SSH protocol implementation, so safe and fast to use
on thousand of nodes.

Expand All @@ -67,7 +69,7 @@ go get github.com/reconquest/orgalorg
# Example usages

`-o <host>...` in later examples will mean any supported combination of
host-specification arguments, like
host-specification arguments, like
`-o node1.example.com -o node2.example.com`.

## Evaluating command on hosts in parallel
Expand Down Expand Up @@ -170,7 +172,6 @@ So, orgalorg expected to work with third-party synchronization tool, that
will do actual files relocation and can be quite intricate, **but orgalorg can
work without that tool and perform simple files sync (more on this later)**.


## Global Cluster Lock

Before doing anything else orgalorg will perform global cluster lock. That lock
Expand All @@ -191,7 +192,6 @@ sync procedure.
User can stop there by using `--lock` or `-L` flag, effectively transform
orgalorg to the distributed locking tool.


## File Upload

Files will be sent from local node to the amount of specified nodes.
Expand All @@ -209,7 +209,8 @@ directory and then exit.

orgalorg preserves all file attributes while transfer as well as user and group
IDs. That behaviour can be changed by using `--no-preserve-uid` and
`--no-preseve-gid` command line options.
`--no-preseve-gid` command line options. These flags are ignored when orgalorg
is ran from Windows.

By default, orgalorg will keep source file paths as is, creating same directory
layout on the target nodes. E.g., if orgalorg told to upload file `a` while
Expand All @@ -224,7 +225,6 @@ remote nodes, `--sudo` or `-x` can be used to elevate to root before uploading
files. It makes possible to login to the remote nodes under normal user and
rewrite system files.


## Synchronization Tool

After file upload orgalorg will execute synchronization tool
Expand All @@ -245,7 +245,6 @@ to that program by using `--stdin` or `-i` flag.
Tool can accept number of arguments, which can be specified by using `-g` or
`--arg` flags.


# Synchronization Protocol

orgalorg will communicate with given sync tool using special sync protocol,
Expand All @@ -264,7 +263,6 @@ given prefix will be printed as is, untouched.

Communication begins from the hello message.


## Protocol

### HELLO
Expand Down
94 changes: 0 additions & 94 deletions archive.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,10 @@ package main

import (
"archive/tar"
"fmt"
"io"
"os"
"path/filepath"
"sync"
"syscall"

"github.com/reconquest/hierr-go"
"github.com/reconquest/lineflushwriter-go"
Expand Down Expand Up @@ -154,98 +152,6 @@ func archiveFilesToWriter(
return nil
}

func writeFileToArchive(
fileName string,
stream io.Writer,
archive *tar.Writer,
workDir string,
preserveUID, preserveGID bool,
) error {
fileInfo, err := os.Stat(fileName)
if err != nil {
return hierr.Errorf(
err,
`can't stat file for archiving: '%s`, fileName,
)
}

// avoid tar warnings about leading slash
tarFileName := fileName
if tarFileName[0] == '/' {
tarFileName = tarFileName[1:]

fileName, err = filepath.Rel(workDir, fileName)
if err != nil {
return hierr.Errorf(
err,
`can't make relative path from: '%s'`,
fileName,
)
}
}

header := &tar.Header{
Name: tarFileName,
Mode: int64(fileInfo.Sys().(*syscall.Stat_t).Mode),
Size: fileInfo.Size(),

ModTime: fileInfo.ModTime(),
}

if preserveUID {
header.Uid = int(fileInfo.Sys().(*syscall.Stat_t).Uid)
}

if preserveGID {
header.Gid = int(fileInfo.Sys().(*syscall.Stat_t).Gid)
}

tracef(
hierr.Errorf(
fmt.Sprintf(
"size: %d bytes; mode: %o; uid/gid: %d/%d; modtime: %s",
header.Size,
header.Mode,
header.Uid,
header.Gid,
header.ModTime,
),
`local file: %s; remote file: %s`,
fileName,
tarFileName,
).Error(),
)

err = archive.WriteHeader(header)

if err != nil {
return hierr.Errorf(
err,
`can't write tar header for fileName: '%s'`, fileName,
)
}

fileToArchive, err := os.Open(fileName)
if err != nil {
return hierr.Errorf(
err,
`can't open fileName for reading: '%s'`,
fileName,
)
}

_, err = io.Copy(stream, fileToArchive)
if err != nil {
return hierr.Errorf(
err,
`can't copy file to the archive: '%s'`,
fileName,
)
}

return nil
}

func getFilesList(relative bool, sources ...string) ([]file, error) {
files := []file{}

Expand Down
106 changes: 106 additions & 0 deletions archive_write_unix.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
//go:build !windows

package main

import (
"archive/tar"
"fmt"
"io"
"os"
"path/filepath"
"syscall"

"github.com/reconquest/hierr-go"
)

func writeFileToArchive(
fileName string,
stream io.Writer,
archive *tar.Writer,
workDir string,
preserveUID, preserveGID bool,
) error {
fileInfo, err := os.Stat(fileName)
if err != nil {
return hierr.Errorf(
err,
`can't stat file for archiving: '%s`, fileName,
)
}

// avoid tar warnings about leading slash
tarFileName := fileName
if tarFileName[0] == '/' {
tarFileName = tarFileName[1:]

fileName, err = filepath.Rel(workDir, fileName)
if err != nil {
return hierr.Errorf(
err,
`can't make relative path from: '%s'`,
fileName,
)
}
}

header := &tar.Header{
Name: tarFileName,
Mode: int64(fileInfo.Sys().(*syscall.Stat_t).Mode),
Size: fileInfo.Size(),

ModTime: fileInfo.ModTime(),
}

if preserveUID {
header.Uid = int(fileInfo.Sys().(*syscall.Stat_t).Uid)
}

if preserveGID {
header.Gid = int(fileInfo.Sys().(*syscall.Stat_t).Gid)
}

tracef(
hierr.Errorf(
fmt.Sprintf(
"size: %d bytes; mode: %o; uid/gid: %d/%d; modtime: %s",
header.Size,
header.Mode,
header.Uid,
header.Gid,
header.ModTime,
),
`local file: %s; remote file: %s`,
fileName,
tarFileName,
).Error(),
)

err = archive.WriteHeader(header)

if err != nil {
return hierr.Errorf(
err,
`can't write tar header for fileName: '%s'`, fileName,
)
}

fileToArchive, err := os.Open(fileName)
if err != nil {
return hierr.Errorf(
err,
`can't open fileName for reading: '%s'`,
fileName,
)
}

_, err = io.Copy(stream, fileToArchive)
if err != nil {
return hierr.Errorf(
err,
`can't copy file to the archive: '%s'`,
fileName,
)
}

return nil
}
Loading