Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LVM module #184

Merged
merged 71 commits into from
Nov 10, 2016
Merged
Changes from 1 commit
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
d211570
[WIP] LVM proof-of-concept
avnik Aug 22, 2016
5d4adc7
lvm: stub for new TastStatus interface
avnik Aug 26, 2016
6763f1f
lvm: rename abbreviated LV/PV/VG to full names
avnik Aug 26, 2016
024c3b2
lvm: refactor
avnik Aug 29, 2016
825165e
lvm: add few tests with real data
avnik Sep 6, 2016
7b293e4
lvm: filesystem formatter/mounter
avnik Sep 12, 2016
fc1274e
lvm: LV creation
avnik Sep 13, 2016
3dc8b7a
lvm: adapt to new modules registering scheme
avnik Sep 13, 2016
190a5ea
more tests
avnik Sep 15, 2016
e4d4cac
lvm: new Check/Apply interface
avnik Sep 19, 2016
96020c5
lvm: Add DI for test {Read,Write}File
avnik Sep 21, 2016
989ca68
lvm: push systemd details into LVM object
avnik Sep 21, 2016
0e55d96
lvm: Add VG highlevel test
avnik Sep 26, 2016
20a5c26
lvm: add basic test for LV and FS
avnik Sep 26, 2016
ad1c166
lvm: Add MakeAll to executor
avnik Sep 29, 2016
24cd9fe
lvm: adopt latest interface changes
avnik Sep 29, 2016
ef37bb3
lvm: refactor LV module after real-hw testing
avnik Sep 30, 2016
f39084b
lvm: workaround for `blkid on device, not created yet`
avnik Oct 3, 2016
d2e7ebc
lvm: more workarounds for VG, not created yet
avnik Oct 3, 2016
4931cb9
lvm: avoid removing of unrelated volumes from VG
avnik Oct 3, 2016
8b41ffb
lvm: typo
avnik Oct 6, 2016
f9da083
lvm: Vagrantfile and hcl for initial testing
avnik Oct 6, 2016
802b1c9
lvm: assorted fixes
avnik Oct 14, 2016
354bdfb
lvm: fix last failing test
avnik Oct 17, 2016
9d6cae5
lvm: add mockable calls for Getuid and Lookup
avnik Oct 19, 2016
5ff730e
lvm: add test for root and required tools to all modules
avnik Oct 19, 2016
9993990
lvm: unpublish OsExec structure
avnik Oct 19, 2016
3547267
lvm: remove abstraction leaks (-2 fixmes)
avnik Oct 19, 2016
08531f5
lvm/fs: make resourceFS a private structure
avnik Oct 19, 2016
1c2a724
lvm/vg: use EvalSymlinks from standard library
avnik Oct 19, 2016
70d3a2a
lvm: make realLVM private structure, as well as .backend field in it
avnik Oct 20, 2016
c6302ec
lvm: attempt to reformat code to make CC happy
avnik Oct 24, 2016
ea4701b
lvm: refactor size parsing and LV preparer
avnik Oct 24, 2016
9cd1156
lvm/lowlevel: Reformat tests using t.Run
avnik Oct 24, 2016
e5069c6
lvm: more tests for lowlevel.*
avnik Oct 24, 2016
e94ada0
lvm/fs: small refactor to reduce lint errors
avnik Oct 24, 2016
93e9825
lvm: silly cc dislike *me receiver which is really abbreviation
avnik Oct 24, 2016
eafd054
lvm: re-format more tests, using t.Run()
avnik Oct 24, 2016
1c11423
lvm/testhelpers: comment all function in mock.go
avnik Oct 24, 2016
1cdd2a1
lvm/vg: refactoring, fix CC complains
avnik Oct 24, 2016
de52e69
lvm: assorted codeclimate fixups
avnik Oct 24, 2016
9b5b10a
Makefile: add 'make fmt' target
avnik Oct 26, 2016
a897927
lvm: parametrize this simple example
avnik Oct 26, 2016
f94298d
lvm/vg: Add more tests for VG
avnik Oct 31, 2016
ebf44fd
lvm: Add license header
avnik Oct 31, 2016
edd1fcc
lvm: more assorted fixes from code review
avnik Oct 31, 2016
0ff6b37
lvm: replace all remaining FIXMEs with NB, and reference to issue
avnik Oct 31, 2016
41108a8
lvm: remove some unused code
avnik Oct 31, 2016
d7e7c18
lvm: demote device existence check to low-level
avnik Oct 31, 2016
e754c1d
lvm: whitespace in comments
avnik Oct 31, 2016
baa0b04
lvm: missing comments for vg_test
avnik Oct 31, 2016
11add34
lvm: fix failing test, and reenable it
avnik Nov 1, 2016
0b57634
lvm/vg: Documet preparer, add `remove` option
avnik Nov 2, 2016
dfa61fc
lvm/lv: LV preparer documentation
avnik Nov 3, 2016
a8d27b2
lvm/fs: Document preparer
avnik Nov 3, 2016
1377e85
lvm: rename resources to full names
avnik Nov 3, 2016
1c813cf
lvm/vg: More tests
avnik Nov 3, 2016
caca3ec
lvm: avoid using braces for variable scoping
avnik Nov 3, 2016
acc71b2
lvm: more tests
avnik Nov 7, 2016
0e4665a
lvm: enable documentation
avnik Nov 7, 2016
d1e2d57
lvm: update to use new api with context.Context
avnik Nov 7, 2016
7f592d0
lvm: more error wrapping
avnik Nov 7, 2016
23feae8
lvm: fix CC warnings
avnik Nov 7, 2016
f2ef449
fix preparers
Nov 8, 2016
8a35f18
add preparer tests
Nov 8, 2016
19c98dc
mock symlink evaluation
Nov 10, 2016
d97a1df
comment test funcs for codeclimate
Nov 10, 2016
6c61d44
use nil instead of empty string or 0 for retry
Nov 10, 2016
11be508
name context in preparers
Nov 10, 2016
59f7bef
prune dead code
Nov 10, 2016
564500c
linter fixes
Nov 10, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
mock symlink evaluation
Rebecca Skinner committed Nov 10, 2016
commit 19c98dc6dc374f28bc891d6585a9bf3368e05025
8 changes: 8 additions & 0 deletions resource/lvm/lowlevel/exec.go
Original file line number Diff line number Diff line change
@@ -19,6 +19,7 @@ import (
"io/ioutil"
"os"
"os/exec"
"path/filepath"
"strings"
"syscall"

@@ -44,6 +45,9 @@ type Exec interface {
WriteFile(fn string, c []byte, p os.FileMode) error
MkdirAll(path string, perm os.FileMode) error
Exists(path string) (bool, error)

// Local Filesystem Functions
EvalSymlinks(string) (string, error)
}

type osExec struct {
@@ -54,6 +58,10 @@ func MakeOsExec() Exec {
return &osExec{}
}

func (*osExec) EvalSymlinks(path string) (string, error) {
return filepath.EvalSymlinks(path)
}

func (*osExec) Run(prog string, args []string) error {
log.WithField("module", "lvm").Infof("Executing %s: %v", prog, args)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are trying to avoid using log. This will not be displayed to the user, so you should use status or an error. If you do need to log, use debug level. Please fix throughout.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address using log everywhere: if the user needs to see the information, it should be changed to an error or status message. We are trying to avoid using log, but if it should stay as a log, please change it to debug level (instead of info). Please fix in the other locations as well.

e := exec.Command(prog, args...).Run()
55 changes: 49 additions & 6 deletions resource/lvm/lowlevel/util.go
Original file line number Diff line number Diff line change
@@ -16,6 +16,7 @@ package lowlevel

import (
"fmt"
"path/filepath"

"github.com/asteris-llc/converge/resource/wait"
"github.com/pkg/errors"
@@ -66,28 +67,52 @@ func MakeRealLVM(backend Exec) LVM {

func (lvm *realLVM) CreateVolumeGroup(vg string, devs []string) error {
args := []string{vg}
args = append(args, devs...)
var canonicalDevs []string
for _, dev := range devs {
canonicalDev, err := lvm.backend.EvalSymlinks(dev)
if err != nil {
return errors.Wrap(err, "resolving symlink for "+dev)
}
canonicalDevs = append(canonicalDevs, canonicalDev)
}
args = append(args, canonicalDevs...)
return lvm.backend.Run("vgcreate", args)
}

func (lvm *realLVM) ExtendVolumeGroup(vg string, dev string) error {
return lvm.backend.Run("vgextend", []string{vg, dev})
canonicalDev, err := lvm.backend.EvalSymlinks(dev)
if err != nil {
return err
}
return lvm.backend.Run("vgextend", []string{vg, canonicalDev})
}

func (lvm *realLVM) ReduceVolumeGroup(vg string, dev string) error {
return lvm.backend.Run("vgreduce", []string{vg, dev})
canonicalDev, err := lvm.backend.EvalSymlinks(dev)
if err != nil {
return err
}
return lvm.backend.Run("vgreduce", []string{vg, canonicalDev})
}

func (lvm *realLVM) CreatePhysicalVolume(dev string) error {
return lvm.backend.Run("pvcreate", []string{dev})
canonicalDev, err := lvm.backend.EvalSymlinks(dev)
if err != nil {
return err
}
return lvm.backend.Run("pvcreate", []string{canonicalDev})
}

func (lvm *realLVM) RemovePhysicalVolume(dev string, force bool) error {
canonicalDev, err := lvm.backend.EvalSymlinks(dev)
if err != nil {
return err
}
args := []string{}
if force {
args = append(args, "--force", "--force", "--yes")
}
args = append(args, dev)
args = append(args, canonicalDev)
return lvm.backend.Run("pvremove", args)
}

@@ -98,7 +123,11 @@ func (lvm *realLVM) CreateLogicalVolume(group string, volume string, size *LvmSi
}

func (lvm *realLVM) Mkfs(dev string, fstype string) error {
return lvm.backend.Run("mkfs", []string{"-t", fstype, dev})
canonicalDev, err := lvm.backend.EvalSymlinks(dev)
if err != nil {
return err
}
return lvm.backend.Run("mkfs", []string{"-t", fstype, canonicalDev})
}

func (lvm *realLVM) Mountpoint(path string) (bool, error) {
@@ -153,3 +182,17 @@ func (lvm *realLVM) WaitForDevice(path string) error {
}
return nil
}

// evalDeviceSymlinks returns the real path of deach device (otherwise it breaks
// on GCE)
func evalDeviceSymlinks(devices []string) ([]string, error) {
realpaths := make([]string, len(devices))
for idx, dev := range devices {
realpath, err := filepath.EvalSymlinks(dev)
if err != nil {
return realpaths, errors.Wrap(err, "unable to resolve path: "+dev)
}
realpaths[idx] = realpath
}
return realpaths, nil
}
8 changes: 8 additions & 0 deletions resource/lvm/testhelpers/lvm.go
Original file line number Diff line number Diff line change
@@ -15,6 +15,8 @@
package testhelpers

import (
"fmt"

"github.com/asteris-llc/converge/resource/lvm/lowlevel"
"github.com/stretchr/testify/mock"
)
@@ -31,6 +33,12 @@ func MakeFakeLvm() (lowlevel.LVM, *FakeLVM) {
return lvm, lvm
}

// EvalSymlinks mocks symlink evaluation
func (*FakeLVM) EvalSymlinks(s string) (string, error) {
fmt.Println("calling fakelvm eval symlinks with ", s)
return s, nil
}

// Check is mock for LVM.Check()
func (f *FakeLVM) Check() error {
return f.Called().Error(0)
8 changes: 7 additions & 1 deletion resource/lvm/testhelpers/mock.go
Original file line number Diff line number Diff line change
@@ -15,9 +15,10 @@
package testhelpers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing license header


import (
"os"

"github.com/asteris-llc/converge/resource/lvm/lowlevel"
"github.com/stretchr/testify/mock"
"os"
)

// MockExecutor is a lowlevel.Exec impleentation for faking system interoperation
@@ -96,3 +97,8 @@ func (mex *MockExecutor) Exists(path string) (bool, error) {
func (mex *MockExecutor) Getuid() int {
return mex.Called().Int(0)
}

// EvalSymlinks mocks symlink evaluation
func (mex *MockExecutor) EvalSymlinks(s string) (string, error) {
return s, nil
}
15 changes: 1 addition & 14 deletions resource/lvm/vg/preparer.go
Original file line number Diff line number Diff line change
@@ -15,8 +15,6 @@
package vg
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing license header


import (
"path/filepath"

"golang.org/x/net/context"

"github.com/asteris-llc/converge/load/registry"
@@ -46,18 +44,7 @@ type Preparer struct {

// Prepare a new task
func (p *Preparer) Prepare(_ context.Context, render resource.Renderer) (resource.Task, error) {
// Device paths need to be real devices, not symlinks
// (otherwise it breaks on GCE)
devices := make([]string, len(p.Devices))
for i, dev := range p.Devices {
var err error
devices[i], err = filepath.EvalSymlinks(dev)
if err != nil {
return nil, err
}
}

rvg := NewResourceVG(lowlevel.MakeLvmBackend(), p.Name, devices, p.Remove, p.ForceRemove)
rvg := NewResourceVG(lowlevel.MakeLvmBackend(), p.Name, p.Devices, p.Remove, p.ForceRemove)
return rvg, nil
}

3 changes: 2 additions & 1 deletion resource/lvm/vg/vg_test.go
Original file line number Diff line number Diff line change
@@ -205,13 +205,14 @@ func TestCreateVolume(t *testing.T) {

me.On("Read", "pvs", mock.Anything).Return("", nil)
me.On("Read", "vgs", mock.Anything).Return("", nil)

me.On("Run", "vgcreate", []string{"vg0", "/dev/sda1"}).Return(nil)

fr := fakerenderer.New()

r := vg.NewResourceVG(lvm, "vg0", []string{"/dev/sda1"}, false, false)
status, err := r.Check(context.Background(), fr)
assert.NoError(t, err)
require.NoError(t, err)
assert.True(t, status.HasChanges())
comparison.AssertDiff(t, status.Diffs(), "vg0", "<not exists>", "/dev/sda1")